BlueSCSI Toolbox API v1 Ideas

[erichelgeson]

The BlueSCSI Toolbox v0 was developed quite a few years ago now - it started in BlueSCSI V1 and it's use has expanded greatly and there are many things we could improve to make it better for developers and users. This discussion is for Toolbox Developers and Users to chime in with ideas on features to add. I'll start with a few I've already done work on. I'll keep this post up to date with accepted features for BlueSCSI Toolbox v1.

Accepted

• Allow for changing of directories in the SD transfer app (non breaking change)
  • This has been coded up - just needs integration tests and a bit more flushing out
• Settings API - Allow users to set properties in the ini file and allow serialization back to the ini file.
  • There is a cache layer setup for the ini so makes updates a bit tricker, but doable.
• Explicitly use all CDB's as 10 bytes. Some are used like 6 byte.
• Avoid LUN prefix in CDB[1] (breaking change)
• 100% integration test coverage - Currently about 90% for the Toolbox
• Expand enumeration API to include details about the emulated device that are not available from inquiry or other SCSI commands.
  • eg: Index of currently mounted CD.
• Include Toolbox API version and firmware version inquiry response (Done)

Considering

• Remove 100 file limit - support up to 255(?)
• Evaluate any commands that could be sub commands once we bump to CBD10 len. (the send file dance?)
• Simplify sendfile?
• Create device API (eg: create new HDD of size X)

[saybur]

If this will already include breaking changes, it might be worth reassigning 0xD8 and 0xD9 to other opcodes to avoid Plextor and Apple CD 300 Plus vendor commands.

[nielsmh]

Yes, I would really prefer if they clash with as few other known vendor commands as possible too.

[saybur]

I've had a chance to dig pretty deep into the toolbox while developing my own app. This comment ended up being quite long (sorry). I hope it doesn't come across as overly critical, the toolbox feature is great and makes it easier to work with SCSI emulators. Here's some thoughts for future versions.

Mode Page Version Negotiation

Rather than making a straight-up breaking change to the API, I’d suggest letting clients choose a compatibility level via the existing mode page. The basic process could look like this:

• Clients pull page 0x31. If offset 41 is greater than 0, the device supports additional toolbox features, (same as the existing plan, as I understand it). However, those features are not yet active: the device will respond to all toolbox commands as it does currently, at API level 0.
• Clients issue MODE SELECT to the page and change some value to indicate they want to upgrade to a new API level. For example, the space character 0x20 after SCSI at offset 8 could be the baseline, and setting it to 0x21 indicates the client is using toolbox API v1.
• The device sees the new byte and changes the toolbox command handler to a new one for API v1. At this point the new command formats are active.

This seems like it has multiple advantages:

• Existing clients are oblivious to new features. The old commands still work fine.
• Updated clients get access to features at a level they can support.
• Future API changes can be made without having another round of incompatibility squashing. Clients ask for what they understand and the device manages it from there.
• API changes can be more granular. Instead of having to do everything at once you can roll out features over time, incrementing the version appropriately.
• If an older client happens to run after a newer client has modified the page, it will see a mode page mismatch and can act accordingly.

The obvious downside is this increases complexity in the firmware and test suite, but IMHO it is probably easier to manage there: distributing new client software or making people update their firmware seems like a harder problem overall.

Improve Interoperability

I’d also suggest changing the recommendations on the wiki about how to read the mode page during validation. I get the idea of making sure it’s safe to send commands in the vendor space, but I would love to see the suggested check be made more generic so other devices could work with software clients without needing to incorporate the BlueSCSI trademark. The validation recommendation could be just a subset of the existing string: look for “SCSI” at offset 4-7 for example (or through offset 8, inclusive of the mode page suggestion above).

To be clear, I'm not asking that the first-party BlueSCSI Toolbox apps be made to work with other devices - that's your code, you're obviously free to do whatever you want with it. This is more a request to make the toolbox API itself less vendor-specific. I like openness and open standards, and this seems like a good opportunity to encourage third-party developers to make apps that work across as wide a variety of devices as possible.

Command Format

For the plan to change to 10-byte commands, it would be ideal if they followed the SCSI spec’s general definition (SPC 4.3.2). You’ve already got the LUN prefix in your list, but reserving control byte 10 and any other unused bits would be good, along with adding allocation length in many commands. This approach could help condense the number of unique opcodes and help with clients on memory-limited systems.

As an example, getting both the file count and the list of files could look like this:

Bit Byte	7	6	5	4	3	2	1	0
0	Op: 0xD0
1	Reserved
2	Reserved (Maybe a future selector for a directory?)
3
4
5
6	Reserved							Img
7	Allocation Length
8
9	Reserved

During DATA IN it could return two bytes of data, indicating the number of entries following, then 40 bytes of data per entry like the current naming format. Users could call it with allocation length set to 2 to fetch the number of entries, then call it again with the allocation length set to the appropriate size (or a smaller size if they are memory limited), retrieving the data. Setting bit 0 of byte 6 would return image data instead of shared directory data.

This kind of approach uses fewer commands, is more SCSI-ish (to me, anyway) and could help make minor upgrades in the future less painful.

[nielsmh]

File transfer/management API

From a high-level perspective, the file transfer commands in the current API are very inconsistent.

Format I would like to see for a new API:

• Establish a "session": A session would consist of a current directory, a cached file list for the current directory, and a handle to an open file.
  • The session ID is included in all subsequent commands.
  • When opening a session, you can specify where it begins on the file system: Either in the generic shared folder, or in the image file folder of a given target. This would unify the "query image file list" for changing media with the file management too.
• Expand file name fields, perhaps to 64, 128, or even 240 characters. OS that don't support that will have to deal in another way.
• Change directory command.
  • Not allowed while a file is open.
  • Changing to a directory reads its file list into a cache immediately.
• Raise the directory entry count limitation to 16 bits.
  • This would make full listings potentially too large, so have the client request ranges via a index and count.
  • The response only consists of directory entry structures, but the first one is a meta-entry, which instead contains the path of the current directory and the count of entries in the current directory.
  • Every following directory entry indicates its index, name, size, and attributes.
  • The file size field being 40 bits is a bit strange, shouldn't it just be expanded to full 64 bits?
• Open file command, which selects or creates a file for either reading or writing, based on a mode flag.
  • When a file is opened, whether for reading, writing, or creation, the response is the full directory entry of the file, including its directory index.
  • Creating a file is done by requesting a special directory index (0xFFFF perhaps) and an open mode flag for "create". The returned directory entry reveals the index it was assigned.
  • Attempting to "create" an existing file index would truncate it to zero bytes, while opening it for writing would keep its data.
• Read and write file block commands, implicitly operating on the open file in the session.
• Rename file command, operating on the open file in the session. Newly created files maybe require a rename command before they can be written to.
• Close file command.