Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix unnecessary header overwrites #253

Open
wants to merge 187 commits into
base: REL_14_STABLE_neon
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
187 commits
Select commit Hold shift + click to select a range
e53d24b
[smgr_api] [community] smgr_api.patch
lubennikovaav May 16, 2021
3bfbae8
[contrib/zenith] contrib_zenith.patch
lubennikovaav May 16, 2021
2f7e01f
[walredo] zenith_wal_redo.patch
lubennikovaav May 16, 2021
3200b62
lastWrittenPageLSN.patch
lubennikovaav May 16, 2021
1f4e2b5
Fix GetPage requests right after replaying CREATE DATABASE
hlinnaka May 21, 2021
f1cd5c6
handle_eviction_of_non_wal_logged_pages.patch
lubennikovaav May 16, 2021
78c598d
[walproposer] wal_proposer.patch
lubennikovaav May 16, 2021
dad11c7
persist_unlogged_tables.patch
lubennikovaav May 16, 2021
79457f9
fix_pg_table_size.patch
lubennikovaav May 16, 2021
0d37741
[walredo] fix_gin_redo.patch
lubennikovaav May 16, 2021
6e94b78
[walredo] fix_brin_redo.patch
lubennikovaav May 16, 2021
3f64084
speculative_records_workaround.patch
lubennikovaav May 16, 2021
a71dc6f
wallog_t_ctid.patch
lubennikovaav May 16, 2021
2c8cb7c
vacuumlazy_debug_stub.patch
lubennikovaav May 16, 2021
7677d32
[test] zenith_test_evict.patch
lubennikovaav May 16, 2021
8184a39
fix_sequence_wallogging.patch
lubennikovaav May 16, 2021
da19cde
Bring back change that got lost in refactoring. silence ReadBuffer_co…
lubennikovaav May 18, 2021
34e7d89
[contrib/zenith] [refer #225] if insert WAL position points at the en…
knizhnik Jun 7, 2021
0be1b1b
[walproposer] Create replication slot for walproposer to avoid loose …
knizhnik Jun 8, 2021
fec5731
[walproposer] Skip absent WAL segment removed by pg_resetwal
lubennikovaav Aug 17, 2021
2488a80
[walproposer] Fix breaking out of WalProposerPoll and WaitEventSetWai…
arssher Jun 11, 2021
5dc9a35
[walproposer] Make it possible to start postgres without reading chec…
knizhnik Jun 17, 2021
822921a
[walproposer] Simplify WL_LATCH_SET testing in the walproposer
ololobus Jul 5, 2021
c5783b9
[walredo] Add basic support for Seccomp BPF mode
funbringer Jun 24, 2021
6b975a6
[smgr_api] [contrib/zenith] 1. Do not call mdinit from smgrinit() bec…
knizhnik Jul 9, 2021
5ad2aab
[walproposer] [contrib/zenith] support zenith_tenant
LizardWizzard Jul 15, 2021
2d4ad90
[walproposer] Remove graceful termination of COPY during walproposer …
arssher Aug 4, 2021
2c5dbdf
[walproposer] [contrib/zenith] [refer #395] Do no align sart replicat…
knizhnik Aug 9, 2021
2f693e6
[test] Add contrib/zenith_test_utils with helpers for testing and deb…
lubennikovaav Aug 10, 2021
d5b54a7
[walproposer] Change condition for triggering recovery
knizhnik Aug 11, 2021
26b19c0
[contrib/zenith] Use authentication token passed as environment varia…
LizardWizzard Aug 4, 2021
6116229
[contrib/zenith] Fix race condition while WAL-logging page, leading t…
hlinnaka Aug 13, 2021
ee4f564
[walproposer] Rework walkeeper protocol to use libpq (#60)
sharnoff Aug 13, 2021
14996ae
Build zenithdb/compute-node:latest in CI (zenithdb/console#125)
ololobus Aug 16, 2021
8d0ab18
zenith_regression_tests.patch
lubennikovaav Aug 23, 2021
2e3ae85
Add test function to flush the shared buffer cache.
hlinnaka Aug 24, 2021
de54e5a
Basic safekeeper refactoring and bug fixing.
arssher Aug 17, 2021
c6c8e33
Rename VCL to epochStartLsn and restart_lsn to truncate_lsn.
arssher Aug 27, 2021
9985388
[refer #27] Implement shared relsize cache to improve zenith performa…
knizhnik Aug 3, 2021
6862e55
Cleanup walproposer changes from #60
sharnoff Aug 31, 2021
951bc22
pass tenant id in connection string to safekeeper
LizardWizzard Aug 31, 2021
686272d
Ask pageserver only with LSN's aligned on record boundary.
kelvich Aug 31, 2021
999db4a
[refer #506] Correctly initialize all fields of WAL page header for f…
knizhnik Sep 1, 2021
5bed2be
Add --sync-safekeepers starting standalone walproposer to sync safeke…
knizhnik Aug 19, 2021
610efb1
Update Dockerfile
ololobus Aug 27, 2021
d4b420d
Fix compiler warnings in walproposer.c
ololobus Sep 10, 2021
12bf0bf
Always advance truncateLsn to commitLsn, keeping it on record boundary.
arssher Sep 9, 2021
83654ed
Minor logging editing.
arssher Sep 10, 2021
d219291
Fix walproposer starting streaming point.
arssher Sep 10, 2021
6ba6889
Mark all recovery messages as received by the donor.
arssher Sep 14, 2021
52ccea9
Don't FATAL in walproposer when EOF arrives in SS_IDLE state.
arssher Sep 14, 2021
91392d3
Optimize walproposer starting streaming point.
arssher Sep 15, 2021
1143ea3
Silence compiler warnings:
hlinnaka Sep 17, 2021
41e00a9
Remove unused functions for reading non-rel pages.
hlinnaka Sep 17, 2021
f5d1d4d
Misc cleanup in the code that communicates with the page server.
hlinnaka Sep 17, 2021
ae2c2d2
Improve the protocol between Postgres and page server.
hlinnaka Sep 17, 2021
aa7dba9
Run 'pgindent' on zenith-specific code in contrib/zenith and in walpr…
hlinnaka Sep 17, 2021
e21c7f8
Fix a badly worded comment
hlinnaka Sep 20, 2021
517d219
Simplify a2e929e by storing starting point in walkeeper itself.
arssher Sep 16, 2021
770546f
Catch walkeeper ErrorResponse in PQgetCopyData
sharnoff Sep 22, 2021
207ebd6
Use buffered I/O for reading commands from stdin.
hlinnaka Sep 24, 2021
214affd
Replace fread() with plain read() and a hand-written buffer.
hlinnaka Sep 30, 2021
8ea9a5d
Store unlogged tables locally, and replace PD_WAL_LOGGED.
hlinnaka Oct 6, 2021
7c1df47
Fix queue cleanup in proposer (#93)
petuhovskiy Oct 14, 2021
e49f138
Support read-only nodes
hlinnaka Oct 19, 2021
c38d759
Remove a drop of syncSafekeepers complexity.
arssher Oct 21, 2021
0ae859c
Fix compiler warning.
arssher Oct 21, 2021
a18aa84
Implement backpressure for compute node to avoid WAL overflow
knizhnik Sep 18, 2021
56a1290
Initialize FSM/VM pages through buffer cache
ololobus Oct 19, 2021
9409140
Turn off back pressure by default
ololobus Oct 21, 2021
f0d67ef
ShutdownConnection instead of ResetConnection in more places.
arssher Oct 21, 2021
3de0121
Handle partial writes to stdout in WAL redo process.
hlinnaka Nov 3, 2021
bf17fe6
Handle keepalives while receiving WAL in recovery.
arssher Nov 8, 2021
f9d1682
Fix truncateLsn update (#101)
petuhovskiy Nov 9, 2021
dce3816
[walproposer] Get rid of SAB_Error after rebase
ololobus Nov 10, 2021
ba79713
Add term history to safekeepers.
arssher Nov 5, 2021
0476da8
Clarify the meaning of StandbyReply LSNs used for backpressure
lubennikovaav Nov 9, 2021
2f9af0c
Use max_replication_apply_lag instead of max_replication_write_lag.
lubennikovaav Nov 11, 2021
a7a4127
Forward pageserver connection string to safekeeper
LizardWizzard Nov 25, 2021
198ac63
Move backpressure throttling to ProcessInterrupts()
lubennikovaav Nov 26, 2021
0ecc14e
Stop building docker images in this repo.
kelvich Dec 15, 2021
145a439
[walproposer] Async WAL append (#105)
petuhovskiy Dec 21, 2021
703b9b8
Fix walsender to work with zenith style standbyReply that sends non-z…
lubennikovaav Dec 22, 2021
dd07782
Do not copy the obsolete apply_conf binary into Docker image
ololobus Dec 25, 2021
3795081
Reorder walproposer code in a more natural order (#112)
petuhovskiy Dec 31, 2021
29e6817
Simplify walproposer code (#114)
petuhovskiy Jan 4, 2022
47cd68a
Don't change propTerm after quorum is acquired (#107)
petuhovskiy Jan 4, 2022
63d15a8
walproposer renames (#116)
petuhovskiy Jan 5, 2022
3588035
Report back-pressure trottling status of backend
knizhnik Dec 29, 2021
595838c
Add max_replication_write_lag
knizhnik Dec 29, 2021
c84ad00
Do not throttle wal sender
knizhnik Jan 10, 2022
27354e7
Don't track acks in walproposer (#119)
petuhovskiy Jan 13, 2022
200e7e0
Silence excessively noisy logging from walproposer.
hlinnaka Jan 14, 2022
5013a30
Extend replication protocol with ZenithFeedback message.
lubennikovaav Dec 20, 2021
f816a4d
Allow to join empty safekeeper to existing cluster (#123)
petuhovskiy Jan 27, 2022
a25c83f
Use local relation cache for smgr_exists
knizhnik Dec 29, 2021
e8fd3b6
Reduce walproposer logging after ca5e7beaf.
arssher Feb 4, 2022
55171b3
Implement cluster size quota for zenith compute node.
lubennikovaav Dec 20, 2021
fb17879
Revert "Use local relation cache for smgr_exists"
arssher Feb 9, 2022
5da70d2
Change the unit of cluster size limit GUC to MB, and other fixes.
hlinnaka Feb 9, 2022
085a631
Improve error handling while connecting to page server.
hlinnaka Feb 9, 2022
3e146b4
Make getpage requests interruptible.
hlinnaka Feb 9, 2022
bd2996b
Fix memory leak of messages received from safekeepers.
hlinnaka Feb 10, 2022
5575578
Initialize pgxactoff for walproposer
knizhnik Feb 11, 2022
9acffbe
Fix compilation warning after 8524105dc.
arssher Feb 14, 2022
f74a9d9
Fix more compiler warnings.
arssher Feb 14, 2022
df7a0a1
Remove dead code in handling ZenithFeedback part of an AppendResponse.
hlinnaka Feb 16, 2022
876296d
Expose reading a relation page at a specific LSN (#131)
antons-antons Feb 18, 2022
2be5289
Add backpressure_lsns() function. Fix zenith feedback processing
lubennikovaav Feb 17, 2022
94601e5
Fix zenith_test_utils linkage on macOS
kelvich Feb 23, 2022
d3ccbc1
Add warning fr unrecgonized GUCs with zenith prefix
knizhnik Feb 21, 2022
4a9cee9
Count WAL flushes in walreceiver (#139)
petuhovskiy Mar 8, 2022
a16ff76
Use local relation cache for smgr_exists
knizhnik Dec 29, 2021
8d98cb0
Populate relsize cache when relation is created.
hlinnaka Mar 10, 2022
a2c2c69
Fix pg_table_size() on a view
hlinnaka Mar 28, 2022
78cfb83
Don't set commitLsn to truncateLsn.
arssher Mar 28, 2022
ec76f4f
Raise fatal error on failed recovery (#147)
petuhovskiy Mar 31, 2022
c2e6401
Enable dumping corrupt WAL segments (#145)
antons-antons Apr 1, 2022
f1834ee
Don't hold walproposer WAL in memory (#141)
petuhovskiy Apr 4, 2022
a3c9478
Add --sysid parameter to initdb
knizhnik Jan 27, 2022
b43e732
Give up connection attempt to safekeeper after timeout.
arssher Apr 15, 2022
14a5bc8
Avoid redundand memory allocation and sycnhronization in walredo (#144)
knizhnik Apr 23, 2022
1acb625
Fix missed include for InRecovery (#149)
knizhnik Apr 25, 2022
c309bac
Avoid "bad syscall 39" on assertion failure in WAL redo process.
hlinnaka Apr 26, 2022
4ce564a
Add error context, if replaying a WAL record fails in WAL redo process.
hlinnaka Apr 26, 2022
dc03bcd
Add WARNING for debugging purposes.
hlinnaka Apr 26, 2022
8a65bd4
Turn Assertion into elog(ERROR), to help with debugging.
hlinnaka Apr 26, 2022
fa222b9
Fix errors in WAL redo about relpersistence mismatch.
hlinnaka Apr 26, 2022
fe1cf28
Don't log 'last written LSN ahead of flushed'.
arssher Apr 26, 2022
8b6fa25
Perform inmem_smgr cleaup after processing each record (#154)
knizhnik Apr 27, 2022
6fd7a41
Avoid extending relation in the WAL redo process.
hlinnaka May 3, 2022
6e49f22
Send timeline_start_lsn in Elected and receive it in VoteResponse mes…
arssher May 1, 2022
0a064cf
Verify basebackup LSN against consensus LSN in walproposer.
arssher May 2, 2022
71ed60b
Implement pg_database_size():
lubennikovaav Jul 23, 2021
0903362
Shut down instance on basebackup LSN mismatch.
arssher May 6, 2022
598d318
Use compute-tools from the new neondatabase Docker Hub repo
ololobus May 9, 2022
b920251
zenith_test_utils extension: add neon_xlogflush()
yeputons May 10, 2022
fa4064e
Reduce noise in the logs from inmem_write()
hlinnaka May 13, 2022
7fb8953
Use compute_ctl instead of zenith_ctl (#162)
ololobus May 18, 2022
4f970da
Improve error messages on seccomp loading errors.
hlinnaka May 24, 2022
383181f
Rename contrib/zenith to contrib/neon. Rename custom GUCs:
lubennikovaav May 26, 2022
7913ea5
Rename 'wal_acceptors' GUC to 'safekeepers'
lubennikovaav Apr 19, 2022
fdc02cd
Fix basebackup LSN comparison in walproposer.
arssher May 31, 2022
ae0afcc
Allow specifiyng the different compute-tools tag (#170)
ololobus May 31, 2022
5afaed3
Do not allocate shared memory for wal_redo process (#165)
knizhnik Jun 6, 2022
eeaaead
Add check for NULL for malloc in InternalIpcMemoryCreate (#173)
knizhnik Jun 8, 2022
d027915
Rename ZenithFeedback (#174)
SomeoneToIgnore Jun 10, 2022
fd9a599
Remove callmemaybe bits from compute (#172)
SomeoneToIgnore Jun 10, 2022
ad78fc7
Panic on receiving higher term (#176)
petuhovskiy Jun 22, 2022
0d5f52a
Misc cleanup in libpagestore.c.
hlinnaka May 16, 2022
eaae857
Large last written lsn cache (#177)
knizhnik Jul 9, 2022
118847d
Add uuid-ossp to the supported extensions (#181)
MMeent Jul 12, 2022
b092941
Update last written LSN for gin/gist index metadata (#182)
knizhnik Jul 19, 2022
4c77785
Revert "Update last written LSN for gin/gist index metadata (#182)" (…
knizhnik Jul 26, 2022
9424cb0
Fix `uuid-ossp` build
kelvich Aug 1, 2022
3916814
Remove redundant const (#189)
bojanserafimov Aug 8, 2022
46385c5
Eliminate UnkonwnXLogRecPtr and always use InvalidXLogRecPtr instead …
knizhnik Aug 12, 2022
55cb46e
Init wal redo buffer for fpi (#194)
knizhnik Aug 15, 2022
7444287
Stamp XLP_FIRST_IS_CONTRECORD only if we start writing with page offset.
arssher Aug 14, 2022
f700c2b
Pull 99% of walproposer code into extension. (#188)
MMeent Aug 18, 2022
f733e09
Use ECR for image (#195)
zoete Aug 19, 2022
0c46560
walproposer: do not send pageserver connstring with START_WAL_PUSH (#…
yeputons Aug 25, 2022
9d22aa6
Remove neon extensions from the vendor repo (#197)
MMeent Aug 25, 2022
698ff58
Remove Dockerfile, it's now in the neon repo (#199)
ololobus Aug 26, 2022
5e7bd7d
Move backpressure throttling implementation to neon extension (#203)
knizhnik Sep 2, 2022
4c869e4
Merge last written cache lsn with new main branch (#201)
knizhnik Sep 4, 2022
3d11198
Local prefetch implementation for Postgres 14
knizhnik Sep 12, 2022
63500f4
Disable prefetch by default to make it possible to early merge this P…
knizhnik Sep 12, 2022
59e82e8
Disable prefetch by default to make it possible to early merge this P…
knizhnik Sep 12, 2022
784557d
Update expected output for sysviews test because of changed default v…
knizhnik Sep 12, 2022
eff6e84
Set last written page lsn for created relation (#209)
knizhnik Sep 16, 2022
27a018f
Undo diasming VM check warning in vacuumlazy.c (#213)
knizhnik Sep 20, 2022
04aa290
Pin pages with speculative insert tuples to prevent their reconstruct…
knizhnik Oct 20, 2022
efdcfa3
Fix shared memory initialization for last written LSN cache (#224)
knizhnik Oct 24, 2022
80e5204
Fix upper boundary caculation in the chunks loop in SetLastWrittenLSN…
knizhnik Oct 28, 2022
c236bea
Move walredo process code under pgxn in the main 'neon' repository.
hlinnaka Oct 31, 2022
380877c
Misc cleanup, mostly to reduce unnecessary differences with upstream.
hlinnaka Oct 31, 2022
12aac2d
Optimize prefetch patterns in both heap seqscan and vacuum scans. (#227)
MMeent Nov 7, 2022
38009a2
Fix prefetch issues in parallel scans and vacuum's cleanup scan (#234)
MMeent Nov 11, 2022
da50d99
Use prefetch in pg_prewarm extension (#236)
knizhnik Nov 17, 2022
c6be492
PG14: Prefetch cleanup (#242)
MMeent Nov 23, 2022
8b158b8
Do not produce open file error for unlogged relations (#239)
knizhnik Nov 24, 2022
06edb5a
Maintain last written LSN for each page to enable prefetch on vacuum,…
knizhnik Nov 24, 2022
299bf4f
Prefetch cleanup: (#247)
MMeent Dec 5, 2022
544ec69
Fix uninitialized variable in spgvacuum.c (#250)
MMeent Dec 7, 2022
c22aea6
Update heap pge LSN in case of VM changes even if wal_redo_hints=off …
knizhnik Dec 8, 2022
8465340
fix unnecessary header overwrites
koivunej Dec 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
.git
.vscode
.circleci
tmp_install
compute_build
86 changes: 86 additions & 0 deletions configure
Original file line number Diff line number Diff line change
Expand Up @@ -712,6 +712,7 @@ with_libxml
with_uuid
with_readline
with_systemd
with_libseccomp
with_selinux
with_ldap
with_krb_srvnam
Expand Down Expand Up @@ -858,6 +859,7 @@ with_bsd_auth
with_ldap
with_bonjour
with_selinux
with_libseccomp
with_systemd
with_readline
with_libedit_preferred
Expand Down Expand Up @@ -1564,6 +1566,7 @@ Optional Packages:
--with-ldap build with LDAP support
--with-bonjour build with Bonjour support
--with-selinux build with SELinux support
--with-libseccomp build with libseccomp support
--with-systemd build with systemd support
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
Expand Down Expand Up @@ -8604,6 +8607,39 @@ fi
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_selinux" >&5
$as_echo "$with_selinux" >&6; }

#
# libseccomp
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with libseccomp support" >&5
$as_echo_n "checking whether to build with libseccomp support... " >&6; }



# Check whether --with-libseccomp was given.
if test "${with_libseccomp+set}" = set; then :
withval=$with_libseccomp;
case $withval in
yes)
:
;;
no)
:
;;
*)
as_fn_error $? "no argument expected for --with-libseccomp option" "$LINENO" 5
;;
esac

else
with_libseccomp=no

fi



{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_libseccomp" >&5
$as_echo "$with_libseccomp" >&6; }

#
# Systemd
#
Expand Down Expand Up @@ -14244,6 +14280,56 @@ else
fi


fi

if test "$with_libseccomp" = yes ; then
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for seccomp_init in -lseccomp" >&5
$as_echo_n "checking for seccomp_init in -lseccomp... " >&6; }
if ${ac_cv_lib_seccomp_seccomp_init+:} false; then :
$as_echo_n "(cached) " >&6
else
ac_check_lib_save_LIBS=$LIBS
LIBS="-lseccomp $LIBS"
cat confdefs.h - <<_ACEOF >conftest.$ac_ext
/* end confdefs.h. */

/* Override any GCC internal prototype to avoid an error.
Use char because int might match the return type of a GCC
builtin and then its argument prototype would still apply. */
#ifdef __cplusplus
extern "C"
#endif
char seccomp_init ();
int
main ()
{
return seccomp_init ();
;
return 0;
}
_ACEOF
if ac_fn_c_try_link "$LINENO"; then :
ac_cv_lib_seccomp_seccomp_init=yes
else
ac_cv_lib_seccomp_seccomp_init=no
fi
rm -f core conftest.err conftest.$ac_objext \
conftest$ac_exeext conftest.$ac_ext
LIBS=$ac_check_lib_save_LIBS
fi
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_seccomp_seccomp_init" >&5
$as_echo "$ac_cv_lib_seccomp_seccomp_init" >&6; }
if test "x$ac_cv_lib_seccomp_seccomp_init" = xyes; then :
cat >>confdefs.h <<_ACEOF
#define HAVE_LIBSECCOMP 1
_ACEOF

LIBS="-lseccomp $LIBS"

else
as_fn_error $? "library 'libseccomp' is required for Seccomp BPF support" "$LINENO" 5
fi

fi

# for contrib/uuid-ossp
Expand Down
13 changes: 13 additions & 0 deletions configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -914,6 +914,14 @@ PGAC_ARG_BOOL(with, selinux, no, [build with SELinux support])
AC_SUBST(with_selinux)
AC_MSG_RESULT([$with_selinux])

#
# libseccomp
#
AC_MSG_CHECKING([whether to build with libseccomp support])
PGAC_ARG_BOOL(with, libseccomp, no, [build with libseccomp support])
AC_SUBST(with_libseccomp)
AC_MSG_RESULT([$with_libseccomp])

#
# Systemd
#
Expand Down Expand Up @@ -1564,6 +1572,11 @@ dnl If you want to use Apple's own Bonjour code on another platform,
dnl just add -ldns_sd to LIBS manually.
fi

if test "$with_libseccomp" = yes ; then
AC_CHECK_LIB(seccomp, seccomp_init, [],
[AC_MSG_ERROR([library 'libseccomp' is required for Seccomp BPF support])])
fi

# for contrib/uuid-ossp
if test "$with_uuid" = bsd ; then
AC_CHECK_HEADERS(uuid.h,
Expand Down
34 changes: 34 additions & 0 deletions contrib/pg_prewarm/autoprewarm.c
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
#include "utils/resowner.h"
#include "utils/spccache.h"

#define AUTOPREWARM_FILE "autoprewarm.blocks"

Expand Down Expand Up @@ -433,10 +434,12 @@ void
autoprewarm_database_main(Datum main_arg)
{
int pos;
int io_concurrency;
BlockInfoRecord *block_info;
Relation rel = NULL;
BlockNumber nblocks = 0;
BlockInfoRecord *old_blk = NULL;
BlockInfoRecord *prefetch_blk = NULL;
dsm_segment *seg;

/* Establish signal handlers; once that's done, unblock signals. */
Expand Down Expand Up @@ -502,6 +505,8 @@ autoprewarm_database_main(Datum main_arg)

if (!rel)
CommitTransactionCommand();
else
io_concurrency = get_tablespace_maintenance_io_concurrency(rel->rd_rel->reltablespace);
}
if (!rel)
{
Expand Down Expand Up @@ -536,6 +541,35 @@ autoprewarm_database_main(Datum main_arg)
continue;
}

/* if prefetching is enabled for this relation */
if (io_concurrency > 0)
{
/* make prefetch_blk catch up */
if (blk > prefetch_blk)
{
prefetch_blk = blk;
}

/* now, prefetch all following blocks */
while (prefetch_blk <= &block_info[apw_state->prewarm_stop_idx])
{
/* unless they're of a different relfilenode */
if (prefetch_blk->filenode != blk->filenode ||
prefetch_blk->forknum != blk->forknum ||
prefetch_blk->blocknum >= nblocks)
break;

/* or unless they are more than io_concurrency blocks ahead */
if (blk + io_concurrency <= prefetch_blk)
break;

PrefetchBuffer(rel, prefetch_blk->forknum, prefetch_blk->blocknum);

/* continue with the next block */
prefetch_blk++;
}
}

/* Prewarm buffer. */
buf = ReadBufferExtended(rel, blk->forknum, blk->blocknum, RBM_NORMAL,
NULL);
Expand Down
18 changes: 16 additions & 2 deletions contrib/pg_prewarm/pg_prewarm.c
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,14 @@
#include "access/relation.h"
#include "fmgr.h"
#include "miscadmin.h"
#include "optimizer/cost.h"
#include "storage/bufmgr.h"
#include "storage/smgr.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/spccache.h"

PG_MODULE_MAGIC;

Expand Down Expand Up @@ -184,14 +186,26 @@ pg_prewarm(PG_FUNCTION_ARGS)
}
else if (ptype == PREWARM_BUFFER)
{
BlockNumber prefetch_block = first_block;
Oid nspOid;
int io_concurrency;

nspOid = rel->rd_rel->reltablespace;
io_concurrency = get_tablespace_maintenance_io_concurrency(nspOid);

/*
* In buffer mode, we actually pull the data into shared_buffers.
*/
for (block = first_block; block <= last_block; ++block)
{
Buffer buf;

Buffer buf;
BlockNumber prefetch_stop = block + Min(last_block - block + 1,
io_concurrency);
CHECK_FOR_INTERRUPTS();
while (prefetch_block < prefetch_stop)
{
PrefetchBuffer(rel, forkNumber, prefetch_block++);
}
buf = ReadBufferExtended(rel, forkNumber, block, RBM_NORMAL, NULL);
ReleaseBuffer(buf);
++blocks_done;
Expand Down
1 change: 1 addition & 0 deletions contrib/pg_prewarm/pg_prewarm.control
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ comment = 'prewarm relation data'
default_version = '1.2'
module_pathname = '$libdir/pg_prewarm'
relocatable = true
trusted = true
1 change: 1 addition & 0 deletions src/Makefile.global.in
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,7 @@ with_tcl = @with_tcl@
with_ssl = @with_ssl@
with_readline = @with_readline@
with_selinux = @with_selinux@
with_libseccomp = @with_libseccomp@
with_systemd = @with_systemd@
with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
Expand Down
3 changes: 2 additions & 1 deletion src/backend/access/brin/brin_xlog.c
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ brin_xlog_insert_update(XLogReaderState *record,
}

/* need this page's blkno to store in revmap */
regpgno = BufferGetBlockNumber(buffer);
//ZENITH XXX Don't use BufferGetBlockNumber because wal-redo doesn't pin buffer.
XLogRecGetBlockTag(record, 0, NULL, NULL, &regpgno);

/* insert the index item into the page */
if (action == BLK_NEEDS_REDO)
Expand Down
8 changes: 8 additions & 0 deletions src/backend/access/gin/gininsert.c
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,8 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
elog(ERROR, "index \"%s\" already contains data",
RelationGetRelationName(index));

smgr_start_unlogged_build(index->rd_smgr);

initGinState(&buildstate.ginstate, index);
buildstate.indtuples = 0;
memset(&buildstate.buildStats, 0, sizeof(GinStatsData));
Expand Down Expand Up @@ -408,6 +410,8 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
buildstate.buildStats.nTotalPages = RelationGetNumberOfBlocks(index);
ginUpdateStats(index, &buildstate.buildStats, true);

smgr_finish_unlogged_build_phase_1(index->rd_smgr);

/*
* We didn't write WAL records as we built the index, so if WAL-logging is
* required, write all pages to the WAL now.
Expand All @@ -417,8 +421,12 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
log_newpage_range(index, MAIN_FORKNUM,
0, RelationGetNumberOfBlocks(index),
true);
SetLastWrittenLSNForBlockRange(XactLastRecEnd, index->rd_smgr->smgr_rnode.node, MAIN_FORKNUM, 0, RelationGetNumberOfBlocks(index));
SetLastWrittenLSNForRelation(XactLastRecEnd, index->rd_smgr->smgr_rnode.node, MAIN_FORKNUM);
}

smgr_end_unlogged_build(index->rd_smgr);

/*
* Return statistics
*/
Expand Down
19 changes: 13 additions & 6 deletions src/backend/access/gin/ginxlog.c
Original file line number Diff line number Diff line change
Expand Up @@ -407,6 +407,7 @@ ginRedoSplit(XLogReaderState *record)
rootbuf;
bool isLeaf = (data->flags & GIN_INSERT_ISLEAF) != 0;
bool isRoot = (data->flags & GIN_SPLIT_ROOT) != 0;
XLogRedoAction action;

/*
* First clear incomplete-split flag on child page if this finishes a
Expand All @@ -415,21 +416,27 @@ ginRedoSplit(XLogReaderState *record)
if (!isLeaf)
ginRedoClearIncompleteSplit(record, 3);

if (XLogReadBufferForRedo(record, 0, &lbuffer) != BLK_RESTORED)
action = XLogReadBufferForRedo(record, 0, &lbuffer);
if (action != BLK_RESTORED && action != BLK_DONE)
elog(ERROR, "GIN split record did not contain a full-page image of left page");

if (XLogReadBufferForRedo(record, 1, &rbuffer) != BLK_RESTORED)
action = XLogReadBufferForRedo(record, 1, &rbuffer);
if (action != BLK_RESTORED && action != BLK_DONE)
elog(ERROR, "GIN split record did not contain a full-page image of right page");

if (isRoot)
{
if (XLogReadBufferForRedo(record, 2, &rootbuf) != BLK_RESTORED)
action = XLogReadBufferForRedo(record, 2, &rootbuf);
if (action != BLK_RESTORED && action != BLK_DONE)
elog(ERROR, "GIN split record did not contain a full-page image of root page");
UnlockReleaseBuffer(rootbuf);
if (rootbuf != InvalidBuffer)
UnlockReleaseBuffer(rootbuf);
}

UnlockReleaseBuffer(rbuffer);
UnlockReleaseBuffer(lbuffer);
if (rbuffer != InvalidBuffer)
UnlockReleaseBuffer(rbuffer);
if (lbuffer != InvalidBuffer)
UnlockReleaseBuffer(lbuffer);
}

/*
Expand Down
Loading