Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update pageserver to store multixacts per region #51

Merged
merged 3 commits into from
Sep 14, 2023
Merged

Update pageserver to store multixacts per region #51

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b53d4f2
Update pageserver to store multixacts per region
poojanilangekar Sep 14, 2023
162984b
Fix style and tests
poojanilangekar Sep 14, 2023
3113d48
Changes to fix clippy
poojanilangekar Sep 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions libs/postgres_ffi/src/nonrelfile_utils.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,12 +85,12 @@ pub fn mx_offset_to_member_offset(xid: MultiXactId) -> usize {
+ (xid as u16 % pg_constants::MULTIXACT_MEMBERS_PER_MEMBERGROUP) * 4) as usize
}

fn mx_offset_to_member_page(xid: u32) -> u32 {
xid / pg_constants::MULTIXACT_MEMBERS_PER_PAGE as u32
fn mx_offset_to_member_page(xid: u32, region: u32) -> u32 {
((xid / pg_constants::MULTIXACT_MEMBERS_PER_PAGE as u32) * pg_constants::MAX_REGIONS as u32) + region
}

pub fn mx_offset_to_member_segment(xid: u32) -> i32 {
(mx_offset_to_member_page(xid) / pg_constants::SLRU_PAGES_PER_SEGMENT) as i32
pub fn mx_offset_to_member_segment(xid: u32, region: u32) -> i32 {
(mx_offset_to_member_page(xid, region) / pg_constants::SLRU_PAGES_PER_SEGMENT) as i32
}
Copy link
Collaborator

@ctring ctring Sep 14, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In postgres, there are also MultiXactIdToOffsetPage and MultiXactIdToOffsetSegment macros to locate a multixact id in the offset slru, are there equivalence in the pageserver or are they not used there??

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I think I see it now. In walredo, they inline the formula in the code.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, just for reference, I had made the calls with the same logic here: https://github.com/umd-dslam/postgres/blob/201b091a2f8a5c96c81f8b8a8425ae5e534ed4cb/src/backend/access/transam/multixact.c#L112

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About the walredo -- I was unsure as well, but they have these expected_* variables that they check for segment and offset; maybe that's the reason for computing it without the macros.


// See CSNLogPagePrecedes in csn_log.c
Expand All @@ -117,28 +117,28 @@ mod tests {
// corresponding PostgreSQL C macros (MXOffsetTo*). These test values
// were generated by calling the PostgreSQL macros with a little C
// program.
assert_eq!(mx_offset_to_member_segment(0), 0);
assert_eq!(mx_offset_to_member_page(0), 0);
assert_eq!(mx_offset_to_member_segment(0, 0), 0);
assert_eq!(mx_offset_to_member_page(0, 0), 0);
assert_eq!(mx_offset_to_flags_offset(0), 0);
assert_eq!(mx_offset_to_flags_bitshift(0), 0);
assert_eq!(mx_offset_to_member_offset(0), 4);
assert_eq!(mx_offset_to_member_segment(1), 0);
assert_eq!(mx_offset_to_member_page(1), 0);
assert_eq!(mx_offset_to_member_segment(1, 0), 0);
assert_eq!(mx_offset_to_member_page(1, 0), 0);
assert_eq!(mx_offset_to_flags_offset(1), 0);
assert_eq!(mx_offset_to_flags_bitshift(1), 8);
assert_eq!(mx_offset_to_member_offset(1), 8);
assert_eq!(mx_offset_to_member_segment(123456789), 2358);
assert_eq!(mx_offset_to_member_page(123456789), 75462);
assert_eq!(mx_offset_to_member_segment(123456789, 0), 2358);
assert_eq!(mx_offset_to_member_page(123456789, 0), 4829568);
assert_eq!(mx_offset_to_flags_offset(123456789), 4780);
assert_eq!(mx_offset_to_flags_bitshift(123456789), 8);
assert_eq!(mx_offset_to_member_offset(123456789), 4788);
assert_eq!(mx_offset_to_member_segment(u32::MAX - 1), 82040);
assert_eq!(mx_offset_to_member_page(u32::MAX - 1), 2625285);
assert_eq!(mx_offset_to_member_segment(u32::MAX - 1, 0), 82040);
assert_eq!(mx_offset_to_member_page(u32::MAX - 1, 0), 168018240);
assert_eq!(mx_offset_to_flags_offset(u32::MAX - 1), 5160);
assert_eq!(mx_offset_to_flags_bitshift(u32::MAX - 1), 16);
assert_eq!(mx_offset_to_member_offset(u32::MAX - 1), 5172);
assert_eq!(mx_offset_to_member_segment(u32::MAX), 82040);
assert_eq!(mx_offset_to_member_page(u32::MAX), 2625285);
assert_eq!(mx_offset_to_member_segment(u32::MAX, 0), 82040);
assert_eq!(mx_offset_to_member_page(u32::MAX, 0), 168018240);
assert_eq!(mx_offset_to_flags_offset(u32::MAX), 5160);
assert_eq!(mx_offset_to_flags_bitshift(u32::MAX), 24);
assert_eq!(mx_offset_to_member_offset(u32::MAX), 5176);
Expand Down
14 changes: 10 additions & 4 deletions pageserver/src/walingest.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -944,7 +944,10 @@ impl<'a> WalIngest<'a> {
xlrec: &XlMultiXactCreate,
) -> Result<()> {
// Create WAL record for updating the multixact-offsets page
let pageno = xlrec.mid / pg_constants::MULTIXACT_OFFSETS_PER_PAGE as u32;
let region: u32 = self.timeline.region_id.0 as u32;
let pageno = ((xlrec.mid / pg_constants::MULTIXACT_OFFSETS_PER_PAGE as u32)
* pg_constants::MAX_REGIONS)
+ region;
let segno = pageno / pg_constants::SLRU_PAGES_PER_SEGMENT;
let rpageno = pageno % pg_constants::SLRU_PAGES_PER_SEGMENT;

Expand All @@ -955,6 +958,7 @@ impl<'a> WalIngest<'a> {
NeonWalRecord::MultixactOffsetCreate {
mid: xlrec.mid,
moff: xlrec.moff,
region: region,
},
)?;

Expand Down Expand Up @@ -989,6 +993,7 @@ impl<'a> WalIngest<'a> {
NeonWalRecord::MultixactMembersCreate {
moff: offset,
members: this_page_members,
region: region,
},
)?;

Expand Down Expand Up @@ -1028,9 +1033,10 @@ impl<'a> WalIngest<'a> {
self.checkpoint_modified = true;

// PerformMembersTruncation
let maxsegment: i32 = mx_offset_to_member_segment(pg_constants::MAX_MULTIXACT_OFFSET);
let startsegment: i32 = mx_offset_to_member_segment(xlrec.start_trunc_memb);
let endsegment: i32 = mx_offset_to_member_segment(xlrec.end_trunc_memb);
let region: u32 = self.timeline.region_id.0 as u32;
let maxsegment: i32 = mx_offset_to_member_segment(pg_constants::MAX_MULTIXACT_OFFSET, region);
let startsegment: i32 = mx_offset_to_member_segment(xlrec.start_trunc_memb, region);
let endsegment: i32 = mx_offset_to_member_segment(xlrec.end_trunc_memb, region);
let mut segment: i32 = startsegment;

// Delete all the segments except the last one. The last segment can still
Expand Down
2 changes: 2 additions & 0 deletions pageserver/src/walrecord.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,13 @@ pub enum NeonWalRecord {
MultixactOffsetCreate {
mid: MultiXactId,
moff: MultiXactOffset,
region: u32,
},
/// Extend multixact members SLRU.
MultixactMembersCreate {
moff: MultiXactOffset,
members: Vec<MultiXactMember>,
region: u32
},
/// Mark transaction ID as committed with the given LSN on a CsnLog page
CsnLogSetCommitted {
Expand Down
8 changes: 4 additions & 4 deletions pageserver/src/walredo.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -512,7 +512,7 @@ impl PostgresRedoManager {
transaction_id_set_status(xid, pg_constants::TRANSACTION_STATUS_ABORTED, page);
}
}
NeonWalRecord::MultixactOffsetCreate { mid, moff } => {
NeonWalRecord::MultixactOffsetCreate { mid, moff, region } => {
let (slru_kind, segno, blknum) =
key_to_slru_block(key).or(Err(WalRedoError::InvalidRecord))?;
assert_eq!(
Expand All @@ -523,7 +523,7 @@ impl PostgresRedoManager {
);
// Compute the block and offset to modify.
// See RecordNewMultiXact in PostgreSQL sources.
let pageno = mid / pg_constants::MULTIXACT_OFFSETS_PER_PAGE as u32;
let pageno = ((mid / pg_constants::MULTIXACT_OFFSETS_PER_PAGE as u32)* pg_constants::MAX_REGIONS) + *region;
let entryno = mid % pg_constants::MULTIXACT_OFFSETS_PER_PAGE as u32;
let offset = (entryno * 4) as usize;

Expand All @@ -545,7 +545,7 @@ impl PostgresRedoManager {

LittleEndian::write_u32(&mut page[offset..offset + 4], *moff);
}
NeonWalRecord::MultixactMembersCreate { moff, members } => {
NeonWalRecord::MultixactMembersCreate { moff, members, region } => {
let (slru_kind, segno, blknum) =
key_to_slru_block(key).or(Err(WalRedoError::InvalidRecord))?;
assert_eq!(
Expand All @@ -559,7 +559,7 @@ impl PostgresRedoManager {

// Compute the block and offset to modify.
// See RecordNewMultiXact in PostgreSQL sources.
let pageno = offset / pg_constants::MULTIXACT_MEMBERS_PER_PAGE as u32;
let pageno = ((offset / pg_constants::MULTIXACT_MEMBERS_PER_PAGE as u32) * pg_constants::MAX_REGIONS) + *region;
let memberoff = mx_offset_to_member_offset(offset);
let flagsoff = mx_offset_to_flags_offset(offset);
let bshift = mx_offset_to_flags_bitshift(offset);
Expand Down
Loading