Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use error messages from jcc err.log in experiments #230

Draft
wants to merge 34 commits into
base: main
Choose a base branch
from
Draft

Use error messages from jcc err.log in experiments #230

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
c6f9ef7
extract target error block from err.log
cjx10 Apr 26, 2024
c5b9e41
check err.log in exp
cjx10 Apr 26, 2024
a79c973
skip DWARF error
cjx10 Apr 26, 2024
6b1567b
consider-using-enumerate
cjx10 Apr 26, 2024
2822643
skip clang diag last line
cjx10 Apr 29, 2024
a951227
fix err.log path
cjx10 Apr 29, 2024
9c2a01d
fix binary name
cjx10 Apr 29, 2024
b6a133e
add in all ld error lines
cjx10 Apr 29, 2024
8c72362
fix jcc error message extraction logic
cjx10 Apr 29, 2024
6442609
check end line as start line
cjx10 Apr 29, 2024
24ce372
use err.log errors
cjx10 Apr 29, 2024
3ddd9b8
give detail err.log path
cjx10 Apr 30, 2024
02d08d0
remove color code from lines
cjx10 Apr 30, 2024
f907c6d
extract lld errors
cjx10 Apr 30, 2024
3a32c0d
refine err.log parsing logic
cjx10 Apr 30, 2024
7d748b0
use build log errors when extracting from err.log failed
cjx10 Apr 30, 2024
89e8c67
correspond to changes made in jcc
cjx10 May 1, 2024
fb83573
linker pattern updated for clang-18
cjx10 May 2, 2024
4b92773
add clang output flag pattern
cjx10 May 3, 2024
dcab0e2
add target names to look for in command args
cjx10 May 3, 2024
d42f94d
extend clang error pattern to match with all compiler errors
cjx10 May 3, 2024
2092fa2
only parse output when needed
cjx10 May 3, 2024
cbc7ce9
consider only the last block produced by target but is unexpected error
cjx10 May 3, 2024
20d8a62
clang error pattern in grouping too
cjx10 May 3, 2024
cb4bee8
include end line for consistency
cjx10 May 3, 2024
fca048c
strip color code before checking lines
cjx10 May 3, 2024
e028792
linker error
cjx10 May 3, 2024
05af518
comment for extracting linker error from build log
cjx10 May 3, 2024
c02d7e9
update references to jcc.go
cjx10 May 3, 2024
df1091b
comments
cjx10 May 3, 2024
1656b4b
move linker error parsing to new branch
cjx10 May 7, 2024
be2ce1e
rename
cjx10 May 7, 2024
fd5868c
nits and refactor
cjx10 May 14, 2024
8f4e841
more restructure
cjx10 May 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 39 additions & 9 deletions experiment/builder_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,20 +358,36 @@ def build_and_run_local(
project_target_name = os.path.basename(self.benchmark.target_path)
benchmark_log_path = self.work_dirs.build_logs_target(
benchmark_target_name, iteration)
jcc_errlog_path = self.work_dirs.jcc_errlog_target(benchmark_target_name,
iteration)
build_result.succeeded = self.build_target_local(generated_project,
benchmark_log_path)
# Copy err.log into work dir.
try:
shutil.copyfile(
os.path.join(get_build_artifact_dir(generated_project, "workspace"),
'err.log'),
self.work_dirs.error_logs_target(benchmark_target_name, iteration))
'err.log'), jcc_errlog_path)
except FileNotFoundError as e:
logging.error('Cannot get err.log for %s: %s', generated_project, e)
# Touch err.log in results folder to avoid FileNotFoundError when
# extracting errors.
open(jcc_errlog_path, 'x')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a better solution than this?
I guess this intends to make parsing easier, but it may confuse us to think JCC created an empty file. We will have to search in gcloud logs to distinguish these two cases.

Could we check os.path.isfile() at the beginning extract_error_message() instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a magic string in err.log when err.log does not exist?

This solution is for local experiment, creating an empty file is currently consistent with the behaviour of getting the build log and run log. Might help prevent increasing the complexity of build_and_run() workflow?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a magic string in err.log when err.log does not exist?

This can work but not preferred.
Intuitively, if err.log is not generated, then it should not exist.

creating an empty file is currently consistent with the behaviour of getting the build log and run log

Where do we create empty build log and run logs when they do not exist?

Might help prevent increasing the complexity of build_and_run() workflow?

Not sure if this relates, but I suppose this will only add two lines in code_fixer?

Could we check os.path.isfile() at the beginning extract_error_message() instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do we create empty build log and run logs?

It's in cloud builder, we open the local file object before checking if the file blob on cloud exists
https://github.com/google/oss-fuzz-gen/blob/main/experiment/builder_runner.py#L645

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see; let's keep it, then.
Thanks!

if not build_result.succeeded:
errors = code_fixer.extract_error_message(benchmark_log_path,
project_target_name)
build_result.errors = errors
build_log_errors = code_fixer.extract_error_message(
benchmark_log_path, project_target_name)
jcc_errlog_errors = code_fixer.extract_jcc_errlog_message(
jcc_errlog_path, project_target_name)

# Temp: Compare errors extracted from build log and jcc err.log.
# Use this info to check we found expected errors from err.log
# and improve build log parsing.
if build_log_errors != jcc_errlog_errors:
logging.warning(
'Inconsistent error messages extracted between build.log'
' and err.log for %s', jcc_errlog_path)
# Fallback to use errors parsed from build log when parsing from err.log
# failed.
build_result.errors = jcc_errlog_errors or build_log_errors
return build_result, None

run_result = RunResult()
Expand Down Expand Up @@ -733,7 +749,7 @@ def build_and_run_cloud(
os.path.realpath(target_path), build_log_name)

with open(
self.work_dirs.error_logs_target(generated_target_name, iteration),
self.work_dirs.jcc_errlog_target(generated_target_name, iteration),
'wb') as f:
blob = bucket.blob(err_log_name)
if blob.exists():
Expand All @@ -757,12 +773,26 @@ def build_and_run_cloud(
os.path.realpath(target_path), run_log_name)

if not build_result.succeeded:
errors = code_fixer.extract_error_message(
build_log_errors = code_fixer.extract_error_message(
self.work_dirs.build_logs_target(generated_target_name, iteration),
os.path.basename(self.benchmark.target_path))
build_result.errors = errors
jcc_errlog_errors = code_fixer.extract_jcc_errlog_message(
self.work_dirs.jcc_errlog_target(generated_target_name, iteration),
os.path.basename(self.benchmark.target_path))

# Temp: Compare errors extracted from build log and jcc err.log.
# Use this info to check we found expected errors from err.log
# and improve build log parsing.
if build_log_errors != jcc_errlog_errors:
logging.warning(
'Inconsistent error messages extracted between build.log'
' and err.log for %s',
self.work_dirs.jcc_errlog_target(generated_target_name, iteration))
# Fallback to use errors parsed from build log when parsing from err.log
# failed.
build_result.errors = jcc_errlog_errors or build_log_errors
logging.info('Cloud evaluation of %s indicates a failure: %s',
os.path.realpath(target_path), errors)
os.path.realpath(target_path), build_result.errors)
return build_result, None
logging.info('Cloud evaluation of %s indicates a success.',
os.path.realpath(target_path))
Expand Down
2 changes: 1 addition & 1 deletion experiment/workdir.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ def build_logs_target(self, generated_target_name: str, iteration: int):
return os.path.join(self.build_logs,
f'{generated_target_name}-F{iteration}.log')

def error_logs_target(self, generated_target_name: str,
def jcc_errlog_target(self, generated_target_name: str,
iteration: int) -> str:
return os.path.join(self.build_logs,
f'{generated_target_name}-F{iteration}.err.log')
Expand Down
Loading