Skip to content

implement a feature to allocate an extra word ahead of each individua… #894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

implement a feature to allocate an extra word ahead of each individua… #894

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,9 @@ is_mmtk_object = ["vo_bit"]
# Enable object pinning, in particular, enable pinning/unpinning, and its metadata
object_pinning = []

# Enable allocate extra header of each individual object
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to clarify that VM::EXTRA_HEADER_BYTES will control the exact size of extra header, and there is no control on the alignment of the extra header. We just guarantee there are EXTRA_HEADER_BYTES before the address we return. It could be unaligned access if the binding attempts to write pointers to the extra bytes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this feature only intended for debugging, or can be used in a performance setting?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it is only intended for debugging.

extra_header = []

# The following two features are useful for using Immix for VMs that do not support moving GC.

# Disable any object copying in Immix. This makes Immix a non-moving policy.
Expand Down
15 changes: 14 additions & 1 deletion src/policy/markcompactspace.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ impl<VM: VMBinding> crate::policy::gc_work::PolicyTraceObject<VM> for MarkCompac
}

impl<VM: VMBinding> MarkCompactSpace<VM> {
#[cfg(not(feature = "extra_header"))]
/// We need one extra header word for each object. Considering the alignment requirement, this is
/// the actual bytes we need to reserve for each allocation.
pub const HEADER_RESERVED_IN_BYTES: usize = if VM::MAX_ALIGNMENT > GC_EXTRA_HEADER_BYTES {
Expand All @@ -157,6 +158,18 @@ impl<VM: VMBinding> MarkCompactSpace<VM> {
}
.next_power_of_two();

#[cfg(feature = "extra_header")]
/// We need one extra header word for each object. Considering the alignment requirement, this is
/// the actual bytes we need to reserve for each allocation.
pub const HEADER_RESERVED_IN_BYTES: usize = if VM::MAX_ALIGNMENT > GC_EXTRA_HEADER_BYTES {
VM::MAX_ALIGNMENT + VM::EXTRA_HEADER_BYTES
} else {
GC_EXTRA_HEADER_BYTES + VM::EXTRA_HEADER_BYTES
}
.next_power_of_two();

pub const GC_EXTRA_HEADER_OFFSET: usize = Self::HEADER_RESERVED_IN_BYTES;

// The following are a few functions for manipulating header forwarding poiner.
// Basically for each allocation request, we allocate extra bytes of [`HEADER_RESERVED_IN_BYTES`].
// From the allocation result we get (e.g. `alloc_res`), `alloc_res + HEADER_RESERVED_IN_BYTES` is the cell
Expand All @@ -166,7 +179,7 @@ impl<VM: VMBinding> MarkCompactSpace<VM> {

/// Get the address for header forwarding pointer
fn header_forwarding_pointer_address(object: ObjectReference) -> Address {
object.to_object_start::<VM>() - GC_EXTRA_HEADER_BYTES
object.to_object_start::<VM>() - Self::GC_EXTRA_HEADER_OFFSET
}

/// Get header forwarding pointer for an object
Expand Down
3 changes: 3 additions & 0 deletions src/policy/marksweepspace/native_ms/block.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,9 +285,12 @@ impl Block {
let mut cell = self.start();
let mut last = unsafe { Address::zero() };
while cell + cell_size <= self.start() + Block::BYTES {
#[cfg(not(feature = "extra_header"))]
// The invariants we checked earlier ensures that we can use cell and object reference interchangably
// We may not really have an object in this cell, but if we do, this object reference is correct.
let potential_object = ObjectReference::from_raw_address(cell);
#[cfg(feature = "extra_header")]
let potential_object = ObjectReference::from_raw_address(cell + VM::EXTRA_HEADER_BYTES);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The binding can config EXTRA_HEADER_BYTES, and there is no requirement on what value a binding can use. If a binding uses a value that is not a multiple of the fixed alignment, this code is wrong. You can either introduce some requirements for a 'valid EXTRA_HEADER_BYTES value', or simply change the condition in L264 so we use native_brute_force_sweep if extra_header is enabled.


if !VM::VMObjectModel::LOCAL_MARK_BIT_SPEC
.is_marked::<VM>(potential_object, Ordering::SeqCst)
Expand Down
50 changes: 34 additions & 16 deletions src/util/alloc/bumpallocator.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,25 +61,21 @@ impl<VM: VMBinding> Allocator<VM> for BumpAllocator<VM> {
BLOCK_SIZE
}

#[cfg(not(feature = "extra_header"))]
fn alloc(&mut self, size: usize, align: usize, offset: usize) -> Address {
trace!("alloc");
let result = align_allocation_no_fill::<VM>(self.cursor, align, offset);
let new_cursor = result + size;
self.alloc_impl(size, align, offset)
}

if new_cursor > self.limit {
trace!("Thread local buffer used up, go to alloc slow path");
self.alloc_slow(size, align, offset)
#[cfg(feature = "extra_header")]
fn alloc(&mut self, size: usize, align: usize, offset: usize) -> Address {
let rtn = self.alloc_impl(size + VM::EXTRA_HEADER_BYTES, align, offset);

// Check if the result is valid and return the actual object start address
// Note that `rtn` can be null in the case of OOM
if !rtn.is_zero() {
rtn + VM::EXTRA_HEADER_BYTES
} else {
fill_alignment_gap::<VM>(self.cursor, result);
self.cursor = new_cursor;
trace!(
"Bump allocation size: {}, result: {}, new_cursor: {}, limit: {}",
size,
result,
self.cursor,
self.limit
);
result
rtn
}
}

Expand Down Expand Up @@ -151,6 +147,28 @@ impl<VM: VMBinding> BumpAllocator<VM> {
}
}

fn alloc_impl(&mut self, size: usize, align: usize, offset: usize) -> Address {
trace!("alloc");
let result = align_allocation_no_fill::<VM>(self.cursor, align, offset);
let new_cursor = result + size;

if new_cursor > self.limit {
trace!("Thread local buffer used up, go to alloc slow path");
self.alloc_slow(size, align, offset)
} else {
fill_alignment_gap::<VM>(self.cursor, result);
self.cursor = new_cursor;
trace!(
"Bump allocation size: {}, result: {}, new_cursor: {}, limit: {}",
size,
result,
self.cursor,
self.limit
);
result
}
}

fn acquire_block(
&mut self,
size: usize,
Expand Down
80 changes: 49 additions & 31 deletions src/util/alloc/free_list_allocator.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,41 +42,22 @@ impl<VM: VMBinding> Allocator<VM> for FreeListAllocator<VM> {
self.plan
}

// Find a block with free space and allocate to it
#[cfg(not(feature = "extra_header"))]
fn alloc(&mut self, size: usize, align: usize, offset: usize) -> Address {
debug_assert!(
size <= MAX_BIN_SIZE,
"Alloc request for {} bytes is too big.",
size
);
debug_assert!(align <= VM::MAX_ALIGNMENT);
debug_assert!(align >= VM::MIN_ALIGNMENT);
self.alloc_impl(size, align, offset)
}

if let Some(block) = self.find_free_block_local(size, align) {
let cell = self.block_alloc(block);
if !cell.is_zero() {
// We succeeded in fastpath alloc, this cannot be precise stress test
debug_assert!(
!(*self.plan.options().precise_stress
&& self.plan.base().is_stress_test_gc_enabled())
);
#[cfg(feature = "extra_header")]
fn alloc(&mut self, size: usize, align: usize, offset: usize) -> Address {
let rtn = self.alloc_impl(size + VM::EXTRA_HEADER_BYTES, align, offset);

let res = allocator::align_allocation::<VM>(cell, align, offset);
// Make sure that the allocation region is within the cell
#[cfg(debug_assertions)]
{
let cell_size = block.load_block_cell_size();
debug_assert!(
res + size <= cell + cell_size,
"Allocating (size = {}, align = {}, offset = {}) to the cell {} of size {}, but the end of the allocation region {} is beyond the cell end {}",
size, align, offset, cell, cell_size, res + size, cell + cell_size
);
}
return res;
}
// Check if the result is valid and return the actual object start address
// Note that `rtn` can be null in the case of OOM
if !rtn.is_zero() {
rtn + VM::EXTRA_HEADER_BYTES
} else {
rtn
}

self.alloc_slow(size, align, offset)
}

fn alloc_slow_once(&mut self, size: usize, align: usize, offset: usize) -> Address {
Expand Down Expand Up @@ -141,6 +122,43 @@ impl<VM: VMBinding> FreeListAllocator<VM> {
}
}

// Find a block with free space and allocate to it
fn alloc_impl(&mut self, size: usize, align: usize, offset: usize) -> Address {
debug_assert!(
size <= MAX_BIN_SIZE,
"Alloc request for {} bytes is too big.",
size
);
debug_assert!(align <= VM::MAX_ALIGNMENT);
debug_assert!(align >= VM::MIN_ALIGNMENT);

if let Some(block) = self.find_free_block_local(size, align) {
let cell = self.block_alloc(block);
if !cell.is_zero() {
// We succeeded in fastpath alloc, this cannot be precise stress test
debug_assert!(
!(*self.plan.options().precise_stress
&& self.plan.base().is_stress_test_gc_enabled())
);

let res = allocator::align_allocation::<VM>(cell, align, offset);
// Make sure that the allocation region is within the cell
#[cfg(debug_assertions)]
{
let cell_size = block.load_block_cell_size();
debug_assert!(
res + size <= cell + cell_size,
"Allocating (size = {}, align = {}, offset = {}) to the cell {} of size {}, but the end of the allocation region {} is beyond the cell end {}",
size, align, offset, cell, cell_size, res + size, cell + cell_size
);
}
return res;
}
}

self.alloc_slow(size, align, offset)
}

// Find a free cell within a given block
fn block_alloc(&mut self, block: Block) -> Address {
let cell = block.load_free_list();
Expand Down
82 changes: 50 additions & 32 deletions src/util/alloc/immix_allocator.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,41 +65,21 @@ impl<VM: VMBinding> Allocator<VM> for ImmixAllocator<VM> {
crate::policy::immix::block::Block::BYTES
}

#[cfg(not(feature = "extra_header"))]
fn alloc(&mut self, size: usize, align: usize, offset: usize) -> Address {
debug_assert!(
size <= crate::policy::immix::MAX_IMMIX_OBJECT_SIZE,
"Trying to allocate a {} bytes object, which is larger than MAX_IMMIX_OBJECT_SIZE {}",
size,
crate::policy::immix::MAX_IMMIX_OBJECT_SIZE
);
let result = align_allocation_no_fill::<VM>(self.cursor, align, offset);
let new_cursor = result + size;
self.alloc_impl(size, align, offset)
}

if new_cursor > self.limit {
trace!(
"{:?}: Thread local buffer used up, go to alloc slow path",
self.tls
);
if get_maximum_aligned_size::<VM>(size, align) > Line::BYTES {
// Size larger than a line: do large allocation
self.overflow_alloc(size, align, offset)
} else {
// Size smaller than a line: fit into holes
self.alloc_slow_hot(size, align, offset)
}
#[cfg(feature = "extra_header")]
fn alloc(&mut self, size: usize, align: usize, offset: usize) -> Address {
let rtn = self.alloc_impl(size + VM::EXTRA_HEADER_BYTES, align, offset);

// Check if the result is valid and return the actual object start address
// Note that `rtn` can be null in the case of OOM
if !rtn.is_zero() {
rtn + VM::EXTRA_HEADER_BYTES
} else {
// Simple bump allocation.
fill_alignment_gap::<VM>(self.cursor, result);
self.cursor = new_cursor;
trace!(
"{:?}: Bump allocation size: {}, result: {}, new_cursor: {}, limit: {}",
self.tls,
size,
result,
self.cursor,
self.limit
);
result
rtn
}
}

Expand Down Expand Up @@ -194,6 +174,44 @@ impl<VM: VMBinding> ImmixAllocator<VM> {
self.space
}

fn alloc_impl(&mut self, size: usize, align: usize, offset: usize) -> Address {
debug_assert!(
size <= crate::policy::immix::MAX_IMMIX_OBJECT_SIZE,
"Trying to allocate a {} bytes object, which is larger than MAX_IMMIX_OBJECT_SIZE {}",
size,
crate::policy::immix::MAX_IMMIX_OBJECT_SIZE
);
let result = align_allocation_no_fill::<VM>(self.cursor, align, offset);
let new_cursor = result + size;

if new_cursor > self.limit {
trace!(
"{:?}: Thread local buffer used up, go to alloc slow path",
self.tls
);
if get_maximum_aligned_size::<VM>(size, align) > Line::BYTES {
// Size larger than a line: do large allocation
self.overflow_alloc(size, align, offset)
} else {
// Size smaller than a line: fit into holes
self.alloc_slow_hot(size, align, offset)
}
} else {
// Simple bump allocation.
fill_alignment_gap::<VM>(self.cursor, result);
self.cursor = new_cursor;
trace!(
"{:?}: Bump allocation size: {}, result: {}, new_cursor: {}, limit: {}",
self.tls,
size,
result,
self.cursor,
self.limit
);
result
}
}

/// Large-object (larger than a line) bump allocation.
fn overflow_alloc(&mut self, size: usize, align: usize, offset: usize) -> Address {
trace!("{:?}: overflow_alloc", self.tls);
Expand Down
25 changes: 20 additions & 5 deletions src/util/alloc/large_object_allocator.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,18 @@ impl<VM: VMBinding> Allocator<VM> for LargeObjectAllocator<VM> {
false
}

#[cfg(not(feature = "extra_header"))]
fn alloc(&mut self, size: usize, align: usize, offset: usize) -> Address {
let cell: Address = self.alloc_slow(size, align, offset);
// We may get a null ptr from alloc due to the VM being OOM
if !cell.is_zero() {
allocator::align_allocation::<VM>(cell, align, offset)
self.alloc_impl(size, align, offset)
}

#[cfg(feature = "extra_header")]
fn alloc(&mut self, size: usize, align: usize, offset: usize) -> Address {
let rtn = self.alloc_impl(size + VM::EXTRA_HEADER_BYTES, align, offset);
if !rtn.is_zero() {
rtn + VM::EXTRA_HEADER_BYTES
} else {
cell
rtn
}
}

Expand All @@ -65,4 +70,14 @@ impl<VM: VMBinding> LargeObjectAllocator<VM> {
) -> Self {
LargeObjectAllocator { tls, space, plan }
}

fn alloc_impl(&mut self, size: usize, align: usize, offset: usize) -> Address {
let cell: Address = self.alloc_slow(size, align, offset);
// We may get a null ptr from alloc due to the VM being OOM
if !cell.is_zero() {
allocator::align_allocation::<VM>(cell, align, offset)
} else {
cell
}
}
}
6 changes: 6 additions & 0 deletions src/util/alloc/malloc_allocator.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@ impl<VM: VMBinding> Allocator<VM> for MallocAllocator<VM> {
self.plan
}

#[cfg(feature = "extra_header")]
fn alloc(&mut self, _size: usize, _align: usize, _offset: usize) -> Address {
unimplemented!()
}

#[cfg(not(feature = "extra_header"))]
fn alloc(&mut self, size: usize, align: usize, offset: usize) -> Address {
self.alloc_slow(size, align, offset)
}
Expand Down
9 changes: 9 additions & 0 deletions src/vm/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,4 +71,13 @@ where
/// Note that MMTk does not attempt to do anything to align the cursor to this value, but
/// it merely asserts with this constant.
const ALLOC_END_ALIGNMENT: usize = 1;

#[cfg(feature = "extra_header")]
const EXTRA_HEADER_BYTES: usize =
if Self::MAX_ALIGNMENT > crate::util::constants::BYTES_IN_WORD {
Self::MAX_ALIGNMENT
} else {
crate::util::constants::BYTES_IN_WORD
}
.next_power_of_two();
}