Make use of sense of hearing: Listen and Hear

[boholder]

This is a general description of a series of new ideas.

I was lucky enough to have an in-depth conversation with Ni Shuang, an human–computer interaction expert, about possible improvements to be made to this mod. I've written a blog about this conversation but, this blog pales in explaining the theory behind each solution. It is so pale that I don't think translating it into English will give you a better understanding, sorry.

In summary, I wish we could make more use of the user's sense of hearing, which is their strong suit. There are two types of hearing actions: Listen and Hear.

• Listen, corresponding to Look ability, users actively choose to listen to a particular sound.
• Hear, corresponding to See ability, users passively hear sounds in the environment.

We're doing well on Listen so far, Read Crosshair for example. We need to implement more features related to sound effects (like existed POI Scan or Fall Detector). We should even make this the main direction, since most games for the visually impaired are similarly implemented.

Here are some possible directions.

Audio Orientation

You may want to watch this video which contains ideas about "Audio Orientation", one of the ways to help visual impaired users maintain their perception of space. I think this approach would work well in Minecraft, a complex game that has a endless random generated 3D world.

• Memorizable Location #147
• Temporary Beacon #148
• Make boat rowing narratable #156

Information simplification

The interface of this game is designed for people with normal vision and is very informative. Currently our users have to bear a huge cognitive load to play the game, such as memorizing the relationships between discrete blocks. We can reduce users cognitive load by:

1. Highlighting key information
2. Grouping similar information
3. Make abstract information meaningful
4. Automating complex operations

• Area Map (bird eye view surroundings explorer) #109
• Memorizable Location #147

[Trix-Maiden]

I've been here since Shoaib took over development of the mod, back when it was several mods. I seem to remember he tried to add a unique sound effect to indicate ore blocks, but it would not pan in the stereo field, so he had to switch to a pitched-down sound that was already part of the vanilla game. The same is now true for various POI items, and the fall detector. I would love it if the sounds could be replaced by unique sounds for the mod, especially as the POI ones are taken from note-blocks and we might want to use those in future.

I know a lot of things would be tricky to implement, but I also know how audio games work. In many of them, there are looping sounds for everything from doors to ladders to oncoming enemies. I do not know how much lag implementing similar sounds in the mod would create, and I've done my best to memorise where doors and staircases are, but a lot of people have difficulty remembering loads of coordinates. So do I, to be fair. I have a document full of command strings to paste in that will take me to various places.

If you are able to implement any of the following features, I do suggest it all be toggleable, in case it proves unhelpful for some who have learned to manage without such functionality, or who simply enjoy the general vibe of Minecraft, like me.

• A sound for a door, to be played only when you are on the same X/Z axis as it, and there's nothing between you and the door. This will be particularly helpful in large bases, and in villages or other structures with doors. I specify that you should only hear this when you have uninhibited access to said door because it could be annoying to hear door sounds constantly (this is why I have POI turned off for fluid/chests etc, because when you have many many points of interest around you, you can't hear yourself think for the clamour), and yet it would be so very useful to be able to home in on a door when villagers are not opening or closing it, or you just want to get inside your house without checking coordinates all the time. I would also like this to extend to fence gates and trapdoors, though different sounds for them would be preferable. The sounds used to indicate the door's location should not be the same sound you hear when opening or closing said door, because that would be too confusing.
• A sound for mob spawners, unique to that block, so that if you were mining through an area and heard that sound nearby, you would know to look for it.
• With regard to streamlining the gaining of information, I don't know how possible this one is, but an audio alternative to the announcement of a block being powered would be neat. A soft hum emitted by the powered block, for example, not too intrusive, as with big circuits the sounds would add up, but it could be nice to have.
• I have asked about this before, and I don't know how possible it is even now, but knowing, as a blind person, when I'm walking past an opening in the wall to either side of me would be useful. The fall detector is one thing, but trying to navigate within an abandoned mineshaft when you don't know where all the branches are, or a nether fortress or similar structure when you don't know where you can turn away from the corridor you're in, is sometimes a painful process. In many audiogames, there is an audio indicator when there is a gap in the wall you're walking beside, either the sound you hear when next to a wall stopping (the Manamon series), or a sound alert that plays when there is an opening to a path on either side of you (A Hero's Call's radar system, which also has a little sound for when you are once more next to a wall). This is, admittedly, only helpful in enclosed spaces, though people who struggle to find trees might appreciate knowing they were walking past solid blocks to either side of them in the open as well. The sounds would of course have to be panned to left or right of the player depending on the direction they were facing. I do not know how possible this even is, especially since the games I mentioned are RPGs with prebuilt area maps, but I thought I would bring it up anyways.
• • Regarding trees, it might also be helpful to have a sound denoting where trees are, specifically a sound that plays when there are a lot of leaf blocks and wooden logs adjacent to one another. You can see logs and leaves in front of you of course, but when one is a new player who just wants to find a tree and doesn't notice they walked past three of them before they noticed leaves, being able to hear a rustling where a tree is would be very handy. It would also be a helpful indicator when one has chopped down a large part of a tree, but missed some blocks hanging out of reach. One could hear the sound of a treelike configuration and be able to pillar up to finish gathering the resources.
• A background sound for when you're standing somewhere the light level is zero would be useful for those who want to make sure their base is fully lit up, or that they've ensured their mining tunnels/the caves they venture through are safe. You can tell if somewhere's not lit when mobs spawn, but for someone who is playing in Survival in a difficulty above Peaceful, that might not be the best time to find out you forgot to put torches up for the last minute or so. The darkness sound would also apply to outside areas at night, and there would be a happy bright chime sort of sound to indicate that you stepped into the influence of some light sources/placed a light source that impacted the block you were standing on. This sound would be centred on the player's location and not need to be panned as a result.
• This may straddle the line between useful and annoying, but distinct sounds for important blocks such as beds, crafting tables, inactive furnaces and other blocks that a blind player would not want to lose track of, probably folded into the POI feature and like all the rest, able to be toggled. This would mean a lot of unique sounds but would probably make it just that bit less painful to locate one's crafting table without checking coordinates. There are a lot of things in audiogames that are represented by sound sources, but as I said above, these games have prebuilt maps usually, and the indicated objects on the map are in part a reference for the player. In Minecraft, you can have chests in in large groups in a storage system, and it is not as vital to know where they are. Signs, however, might need an indicating sound so you can know when you're walking past one/that you should probably check said sign and make sure it's the one you're looking for.
• I've not done as much with potions, since I have limited ingredients, but I know there is a visual indicator of how much time a potion effect has left, and like the health/hunger bar, knowing when your potion is about to run out would help a great deal. It would also be beneficial if each potion effect had a different sound to indicate when it would wear off, since you can be under the effects of several potions at a time, and if you had an invisibility potion and a water-breathing potion going at the same time, you'd want to be sure which was running out so you didn't waste any backup potions. I wouldn't want the sound to be the same as health/hunger being low (I'd honestly like it if that sound was different than it is now, but it does its job).
• Not a new feature, but can we please have a clearer sound for items that need to be collected? The current one is audible under some conditions, but when it's raining or music is playing it can be drowned out, and although one could turn off weather sounds or music, I personally like both the soundtrack and hearing the rain, and don't want to sacrifice either of them for the chance to hear items better.

All of these are things a fully sighted person would be able to notice immediately, as even if they're facing one direction, sighted people can see the land around them on the screen, and also obviously know when the light level is low.

Minecraft does come with its own sounds for mobs and relevant entities, as well as various actions and environmental features. All of these do help a lot, though as the above list shows, there are still some things that could, if implemented, improve the experience of totally blind players like myself.

There are also some features which I have not mentioned, because I have found other mods that address these particular concerns. Presence Footsteps makes step sounds nice and loud, which is helpful for navigation (and incidentally makes it so Creepers are audible, which is rather important to totally blind players who do not want their things blown up). Ambient Sounds and MAmbience both provide environmental ambiences for each biome, as well as audible indications of the day/night cycle, and each have their positives and negatives. I personally use Ambient Sounds for its seamless loops, but MAmbience also had sounds for the player moving around in different types of armour (distinctive sounds for leather, iron and diamond armour at least, and probably gold and Netherite as well, though I have never had netherite anything and didn't check with gold). It would be nice if all these features could be folded into a single mod, but not even sighted players can have all their favourite mods folded into a single one, and make do with modpacks or the like. I'm fine using these extra mods for these particular features, although I do wish I could have the armour sounds from MAmbience and the seamless loops from Ambient Sounds at the same time.

I may add other things to this comment as I think of them, but I do not expect everything to be implemented in the exact way I described above. For all I know, you could come up with a much better way of doing things than I described, similar to when you came up with the area map feature, which I look forward to very much, as it sounds like it will be immensely useful!

[boholder]

@Trix-Maiden You've posted so many I can't respond to them all, but I've read them carefully and kept them in memory. Thanks again for so many suggestions!

but it would not pan in the stereo field

If the "pan" means "direction perceivable sound", we can simply achieve this by playing sound at target position, thanks to Minecraft's 3D nature you'll here a stereo. The Fall Detector is implemented in this way.

I would love it if the sounds could be replaced by unique sounds for the mod, especially as the POI ones are taken from note-blocks and we might want to use those in future.

I'm not sure if you mean adding new sounds effects into the game, but the note block has 16 different instrumental tones, more than the type number (11) of ore blocks at least. We can assign different tones to different ore blocks, in POI Scan.

I've done my best to memorise where doors and staircases are, but a lot of people have difficulty remembering loads of coordinates.

I personally want to expand POI Scan or add a actively hold-to-hear key, so you can hear the sound form the special blocks like beds, doors and trap doors. As you said, since Minecraft won't allow mods to creature other threads or processes (is that ture?), heavy logic can cause player-perceptible lag. We must not use some realtime one by one block scan like Fall Detector or Find Closest Fluid.

Here is a hasty and potential way:

1. Record special blocks position silently when Read Crosshair (or Area Map) hits them.
2. Make the data persistent, like saving them into a json file, and load them into memory while the game is starting.
3. Use some bisection index way to gourp positions into small groups, only keep one group active at a time, so we can match less candidates in every game event loop.
4. Write other logic in normal way.

I do suggest it all be toggleable.

Yup, will do so.