Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests for testing correct implementation? #13

Open
madig opened this issue Jan 9, 2019 · 6 comments
Open

Tests for testing correct implementation? #13

madig opened this issue Jan 9, 2019 · 6 comments
Labels
good first issue Good for newcomers

Comments

@madig
Copy link

madig commented Jan 9, 2019

Hi!
First, thanks for the guide, I enjoyed it 😃 I am wondering if there are test-images or something that extensively test if things are implemented correctly? The code I came up with does not match your example code exactly and I'd like to see if would make a difference in a hypothetical "compliance test suite".

@justinmeiners
Copy link
Owner

justinmeiners commented Jan 10, 2019

Other than the two games we don't have a test suite. It is a good idea though since there are a few instructions and trap calls we do not use.

One thought I have about writing such a suite is that it needs to be careful about what it decides to enforce. My LC-3 does not follow the LC-3 specification. For example trap codes are not implemented in assembly, and so they behave internally differently. One could write a program that depended on this behavior and it would not work.

We want to encourage flexibility of implementation, with a reasonable expectation of program compatibility. With this in mind I think the test suite should just verify the basic operations of the trap codes and instructions.

@rpendleton are you interested in this?

@rpendleton
Copy link
Collaborator

I'm interested in this. I don't know exactly how we'd want to implement these tests (comparing output? or writing a test runner in LC-3?) but I'll look into it.

@justinmeiners
Copy link
Owner

justinmeiners commented Apr 21, 2020

I now have an opinion on implementing tests.
The tests should just be an LC-3 program which tries to use each instruction and mode and outputs some results. We can then write a simple shell script to verify output is correct. Of course, failure to pass tests will probably result in things like infinite loops.

Is anyone interested in writing an LC-3 program like this? Perhaps there is one out there?

@justinmeiners
Copy link
Owner

justinmeiners commented Jul 27, 2020

We want to do this. Here is what I want

  • Create a src/test.asm program in lc3 assembly which tests basic operations of each instruction and variant based on the specification included in the supplies.
  • At each test step make logs to stdout using Puts which demonstrate that the test is working.
    For example, addition might add 2 numbers, and print the resulting sum.
  • Create a phony makefile target which runs each unix variant of the lc3 program and uses
    cmp(1) to test that stdout matches a previously verified implementation.
  • Add some comments to the makefile explaining that this test does.
  • Change the end of the tutorial to mention that the test is available.

@justinmeiners justinmeiners added the good first issue Good for newcomers label Jul 27, 2020
@fizaan
Copy link

fizaan commented May 7, 2022

"..trap codes are not implemented in assembly, and so they behave internally differently. One could write a program that depended on this behavior and it would not work."

Hmm..this program when run on the official LC-3 simulator will never HALT (unless R7 is saved first then restored). But it does halt when tested in this implementation of the lc3-vm.

.orig x3000

jsr print
halt

print
 ; st r7, temp
 ld r0, char 
 out
 ; ld r7, temp
 ret

char   .stringz "5"
temp   .fill x0000
.end

@rpendleton
Copy link
Collaborator

rpendleton commented May 8, 2022

Hmm..this program when run on the official LC-3 simulator will never HALT (unless R7 is saved first then restored). But it does halt when tested in this implementation of the lc3-vm.

@fizaan Good catch. I've created #52 to fix this and will have a PR up soon.

Even though we're implementing the TRAPs directly in the VM (as opposed to relying on the assembly-based implementations), we should still be changing R7 as part of the TRAP instruction since that's part of the specification. The fact that R7 doesn't change in our implementation is a pretty obvious difference in behavior and, as you mentioned, leads to programs that work in the article's implementation of the VM but not the official one. So I think it's important to fix.

Repository owner deleted a comment from VinayWolf Mar 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

4 participants