Project Continuation Steps

[mattsta]

So, what do we do now?

As far as I can tell, ib_insync was a single-owner project and now the owner is reportedly dead (who updated their github profile and closed all the projects though?).

We are locked out of the official documentation site and the pypi package and any existing testing infrastructure, so do we:

• rename this as a new project?
• keep the same project but bump a major version number to denote we're making our own changes now?
• should all this live in my own personal account? I have an unused github organization over here we could use too https://github.com/tradelogic
• we also need to audit the ib_insync API version and message compatibility against the current IBKR python API. There's supposedly more message types in newer IBKR gateways not supported in ib_insync yet. Ideally we should be watching for all changes to twsapi updates and making matching modifications to ib_insync
• Eventually it would be nice to have an extra better client than just reproducing the IBKR api directly since it does awful things like use FLOAT_MAX and INT_MAX to mean undefined or null but we should be able to have a proper None-capable python API in more places.

Do we want to keep the same "trunk-branch development" process or use a more interactive next branch where we stage work until it gets tagged as a new version?

I'll start with some minor cleanup (common style formatter, pyproject instead of requirements and setup, etc), but we'll need a longer term plan for adding new features when IBKR APIs change plus continually monitoring for bug fixes along the way.

This project also relies on two of his other projects as utility libraries (eventkit and nest_asyncio) which also present the same problems: locked out of pypi and docs, etc, but I'll go ahead and start updating them with modern style refactoring just to get things moving.

[francdoc]

Hi @mattsta, you raised up some good issues here. You can share these questions in this discussion thread if you want: [link to the thread].

Some experienced developers are actively discussing similar issues (e.g: group organization, future architecture and design optimizations), and their insights could provide valuable guidance.

[salsasepp]

Thank you for your initiative!
I guess you are aware of PEP 541, but anyway: https://peps.python.org/pep-0541/#how-to-request-a-name-transfer
PEP 541 specifies the process of taking over abandoned packages. The process it not trivial and certainly not fast. If successful, continuing the use of the ib_insync package name would be the least disruptive thing IMO. In the meantime, most of us can continue to use the latest pypi release, and you could distribute updated wheels outside of pypi if necessary.

[kulki007]

Hi @mattsta, Could you please respond on the thread linked above. It will be better to come together as a community and decide how we should proceed further. Already ib_insyc is behind the latest IB API.

[mattsta]

The thread is amusing and sad at the same time. What part of dead and [cross] 2024 are people misunderstanding?

Maybe he died and his estate has email/github access and shut things down. [Update: they added a note and that's what happened]

As for group participation, it's an open source curse (combined with the kinds of low knowledge/high maintenance people online trading attracts in general). If tens or hundreds or thousands of people demand software be maintained, but nobody is paying for it, you are at the largesse of generous experienced people to provide valuable work for free.

Feel free to have his sponsors bounce over to my account for more reliable updates. I plan on auditing the current feature difference between the IBKR twsapi package and ib_insync to see what's missing and also to see if we can maintain a cleaner automated update cycle when new TWS/gateway APIs are released.

[kulki007]

Mattsta, I am happy to help you in any way I can.is there anything specific that you need help with that I can provide. Immediate needs seems to be to move to latest 10.27 IB API.

[jlixfeld]

@mattsta - Are you following along in the TWS API thread by chance? I think at the moment there might be yourself and Mel who are trying to make changes to your respective forks, which is great but I wonder if some discussion or collab might be useful? I think this might be a point where a potentially crucial divergence in the code based could stem from and maybe this doesn't have to be the case?

[kulki007]

@mattsta I agree with jixfeld. Instead of having two separate branches it would be better for the community at large to decide which branch to continue to work on. We are also trying to resurrect the ib_insync group on groups.io so that would be a place for people to discuss and ask questions about the API.

[mattsta]

Yeah, according to the ibapi version, the IBKR python API is using server version 180 while current ib_insync is on version 178. The only additions appear to be "fund data fields" and "manual order time exercise options" (added October 2023) in the newer version IDs.

The "17 additional fields for FUND" is currently missing as people mention, but it's a specialized type of mutual fund metadata like this:

class FundAssetType(Enum):                                                                                      
    NoneItem = ("None", "None")                                                                                 
    Others = (("000", "Others"),)                                                                               
    MoneyMarket = ("001", "Money Market")                                                                       
    FixedIncome = ("002", "Fixed Income")                                                                       
    MultiAsset = ("003", "Multi-asset")                                                                         
    Equity = ("004", "Equity")                                                                                  
    Sector = ("005", "Sector")                                                                                  
    Guaranteed = ("006", "Guaranteed")                                                                          
    Alternative = ("007", "Alternative")                                                                        
                                                                                                                
                                                                                                                
class FundDistributionPolicyIndicator(Enum):       
    NoneItem = ("None", "None")               
    AccumulationFund = ("N", "Accumulation Fund")
    IncomeFund = ("Y", "Income Fund") 

Should be easy enough to add. Maybe I'll have time over the next couple days to start clearing up these minor out of date features (or feel free to submit a PR for them too, adding these new FUND fields just requires repetitive template setup copying the logic from tws-api/ibapi/contract.py and tws-api/ibapi/decoder.py to fit the ib_insync data model instead).

Unfortunately due to IBKR's restrictive client API development practices, I can't just link to their usage in a github repo for us to use as a shared reference. I wonder if anybody cares if we create a new repo for the twsapi source too? Is there already one somewhere? They just make everybody click through an annoying EULA to download their zip files without any revision control: https://interactivebrokers.github.io/ — though they do have a "secret CLA repo" somewhere too https://interactivebrokers.github.io/api_software_contribute.html

As for overall continuity of the project, my primary concern is not breaking my personal ibkr order and account management system since it uses ib_insync as the IBKR API, so my priority is maintaining (and improving!) ib_insync code quality and features and performance so my trading interface and algo adapters don't decay over time. I'm less immediately concerned with packaging and doc updates since we are locked out of those anyway for who knows how long (still, assuming this entire "dead status + archive-all-the-things + delete forums" routine isn't just a hack, but is an indication of some final event or actively giving up on everything (or maybe he won a billion hague bucks and just doesn't have time for us anymore!), so we are on our own from now on).

Feel free to open any other topics/issues/PRs here and anybody can talk about anything. Probably a standard pattern of "Issues" for code questions and bug reports, "Discussions" for usage questions as long as we're not doing your homework for you, and "PRs" for, well, you know.

[kulki007]

I agree lets go with ib_async.

[jlixfeld]

How do we feel about insync_edw as the GitHub organization name?

To paraphrase the original contributors from the TWS API list whose recent messages spurred this name:

• insync being included might make it more likely to be found by users who haven’t found this discussion. While it's not in the library name, it's in the organization name so mission accomplished there.
• keeps the original name, recognises the legacy.

[mattsta]

Yeah, it'll be good to keep those references prominent probably more directly in docs and README so they remain more visible (and also builds up more search relevance over time).

I imagine if people searching for a library find the original archived ib_insync repo, they will search for ib_insync replacement somewhere too, so we'll make sure to include those details everywhere so they get picked up easily.

also it would be nice if we could find a copy of the original project announcement forum post from 2017? We'll add a new history/legacy document about how it started and how it's going.

[jlixfeld]

There was somebody who has the entire archive of the original list. We can track them down to provide that information.

[vasiche]

If anybody has all or most/some legacy emails from groups.io I would like to collaborate with that person to see if we can somehow restore or make available to the community those emails. Maybe saving them to a searchable pdf file, as a simple option. There may be a better way, too. Those emails had a lot of helpful information, answers, etc. Let me know if you know who may have those emails.

[kulki007]

@mattsta How exactly does one create a PR if one doesnt have write permissions to the repository? Can anyone create a PR and then submit for approval and then your would approve the PR?

[kulki007]

@mattsta We need to agree on a new name for this project. We intend to create a new subgroup on groups.io but the name of the group should match the new name of this project. Do you any suggestions?

[mattsta]

they temporarily un-archived the repo, wrote an official sad notice, then re-archived it again:

Unfortunately, our dear brother Ewald de Wit passed away monday, March eleven as a result of a serious illness last year.

Although we were not involved in his activities on this platform, we felt the due to inform you. We thank you for knowing him.

looks like we're on our own going forward. at least we can respect the legacy of his years of work by continuing to improve things over time.

[kulki007]

thank you for providing this update. This greatly makes me feel at ease. I wasnt willing to accept that Ewald somehow just went rogue. this makes more sense to me given what I know of him. RIP brother.
I say we just go with a new project. Maybe we acknowledge the contribution that Ewald made by having some name like ib_insync_new or ib_insync_continued or soemthing like that.

[mattsta]

Status for now:

• made new org at https://github.com/ib-api-reloaded — let me know if you want to be added as a helper/contributor and we'll figure out org co-ownership eventually too, but i'll get the first re-branding and renaming steps done before making the new repo public. (again, priority is: repo/code stability, then public packaging, then public docs website)
• It will take a couple days to get the current code renamed and docs updated again. Maybe I'll have more time over the weekend. We'll also need to hope there are never any bugs in his other personal dependencies of eventkit and nest_asyncio, but I'm less concerned with nest_asyncio because it's mostly used for just ipython demos and I think newer jupyter may not even need it anymore.

[jlixfeld]

I think there was a misunderstanding - the consensus was that the org name was to be insync_edw. There seems to be a bunch of support for that on Mel's Discord. I didn't get a chance to communicate that consensus here, apologies.

Also, I'll happily help out with co-ownership if there's any value. I'm not the most seasoned, but I'm happy to learn.

[ReliefCrew]

the consensus was that the org name was to be ...

I didn't notice myself, but I still applaud the initiative. @mattsta... why not send a few invites to the top prospects?

[kulki007]

@mattsta can you set the org name to insync_edw as a tribute to Ewald who worked tirelessly all these years for free and helped so many of us. The actual project can be ib_async like you suggested.

[geepeem]

Great to see some progress! Thanks for spearheading this effort @mattsta . I am happy to help however I can, but am more of a trader than a coder TBH. Let me know if there's anything not too technical that needs doing.
It would be great if we could get a good collaboration going with the IB dev team - it is definitely in their interest to make this project work since it makes using their API a lot simpler for the (probably many) people like me that just sort of picked up python as needed for creating their bots.

[cheukhin1024]

@mattsta Please let me know what I can do. I am working in IB API team.

[kulki007]

one of the issues I face is that we dont have access to a git repository for IB API itself. It will make matters so much easier if we can track changes on your end and then we replicate those changes on our end. Without a source control system, we need to manually diff files to figure out what exactly has changed between versions. Currently the source code is released in a windows installed file format which makes it very clunky to use. I hope you would agree its a lot simpler if it were just on github

[cheukhin1024]

@kulki007 Thank you for your reply. Unfortunately, I cannot make any promises here because I need to follow Interactive Brokers Communication Policy Guidelines and I am not the decision maker.

However, I sent an email to Mr. Matt ([email protected]) using [email protected] and CC API team ( [email protected]) about introducing our new IBKR campus API documentation sites and showing my respect for Mr. Ewald de Wit and his contributions to the ib_insync project.

In my personal opinion, any possible ideas should be sent to Mr. Matt if he is confirmed to be the new main creator and the main contact person for ib_insync 2.0. After that, Mr. Matt can summarize the ideas and email to [email protected] .

[kulki007]

@mattsta I think the general consensus seems to be that we pay our respects to Ewald by creating the new github organization as insync_edw.
We should call the new project as ib_async as you had suggested. If you agree then we can create a Groups.IO account as well under the sub group called ib_async to match the project name.

[mattsta]

Sunday update: over the weekend I renamed everything in the repo from ib_insync to ib_async and modernized some of the repo architecture.

New resources:

• new primary repo: https://github.com/ib-api-reloaded/ib_async (no new package release yet though)
• new docs site: https://ib-api-reloaded.github.io/ib_async/
• place for memories if people would like to add more details: https://github.com/ib-api-reloaded/memorial

Feel free to point out typos or anything else I missed in the refactored repo.

The new org name seems fine since we are making it clear we are continuing the project with all the original details. we are a library of peace. It seems cleaner with an unrelated org name unrelated to the previous activity because even at first glance, it may look like things named edw could still be live activity? please don't name anything after me eventually either.

[geepeem]

This is great work Matt, thanks on behalf of all the people who used ib_insync and hopefully will be able to continue doing so thanks to your efforts!
The new repo looks great, and I agree that it's best to not include edw in the name to avoid any potential confusion. The memorial page is a nice touch.
I will read through the new repo and let you know if I spot anything that's worth changing / correcting.
Going forward, we should probably switch the discussion to the new repo. I will open one to get the ball rolling.
Cheers!

[ClimberMel]

I'm happy to help out with code. Has anyone started on the changes for the fund data? If not, you could assign that one to me. Or if you add me to the repo, I can just assign it to myself and start on it.

[vasiche]

Thank you @mattsta!

[kulki007]

thank you so much @mattsta

[brndnmtthws]

As a suggestion, it might be worthwhile linking to https://github.com/ib-api-reloaded/ib_async at the top of the README and in the repo description/link.

[iamshanvy]

Hey @mattsta

Thank you for creating a fork of repo so quickly, we are building custom solutions for our clients using ib_insync for a while now with some of them having their customised version of the library deployed, feel free to reach us for any help too, we would love to contribute the repo we have necessary resources to do that

Shameless plug, if anyone needs a custom bot developed using ib_insync (or ib_async now) reach us out here

[decoded-markets]

First of all, RIP Edwin. Just came across this thread as I was looking for some info on the ib_insync group and wondered why it was closed... Secondly, thanks heaps @mattsta for the initiative to continue the project. Like many, ib_insync -> ib_async holds a critical place in my automated trading system. I would be more than happy to help contribute. I'm not sure my knowledge of how the library works behind the scene is up to scratch for now... but if push come to shove I'll spend the time it takes to make sure this library keeps on working. There is NO WAY I am starting an integration from scratch given my current IB setup has been working SO PERFECTLY thus far.

[westonplatter]

Looking forward to continuing to use ib_insync from the new ib-api-reloaded source.