Evaluation doesn't work on Windows

[peter-ch]

After getting a score of 0 every time, I looked at the samples.jsonl_results.jsonl file and the result for each is this: "failed: module 'signal' has no attribute 'setitimer'"

This seems like a Windows/Unix issue.

[Ephrem-Adugna]

Same issue here

[mfwong1223]

For Windows, I replaced the signal module by the threading module on

human-eval/human_eval/execution.py

Lines 90 to 99 in 312c5e5

	@contextlib.contextmanager
	def time_limit(seconds: float):
	def signal_handler(signum, frame):
	raise TimeoutException("Timed out!")
	signal.setitimer(signal.ITIMER_REAL, seconds)
	signal.signal(signal.SIGALRM, signal_handler)
	try:
	yield
	finally:
	signal.setitimer(signal.ITIMER_REAL, 0)

to

import threading
@contextlib.contextmanager
def time_limit(seconds: float):
    def signal_handler():
        raise TimeoutException("Timed out!")
    timer = threading.Timer(seconds, signal_handler)
    timer.start()
    try:
        yield
    finally:
        timer.cancel()

[Ephrem-Adugna]

Above didn't work for me, just ran inside linux vm using wsl