Merger with Quantopian trading-calendars

[rsheftel]

Merger with Quantopian Trading-Calendars

Work has begun to merge this project with the Quantopian trading_calendars project that it was originally forked from. The end-state is to have one unified project that brings together the superset of all functionality in this project and trading_calendars, and to have one source for all market and exchange calendars. This will be the trading_calendars project.

The process of this merger will be updated in this issue, everyone is welcome to comment or provide their input

See #41 for the original discussion of the merger. Also see many of the Issues and PR on trading_calendars about this merger.

Why now?

The genesis of the pandas_market_calendars project was to create a version of the Quantopian trading_calendars that was free of all other dependencies. At the time this project was forked the trading-calendars had to be installed with all the other parts of the Quantopian project.

What has changed?

The Quantopian people have removed the unneeded dependencies from trading_calendars and it is now a stand alone project. It is now only a few pure python dependencies, unlinked from the rest of the Quantopian packages. They have changed the one thing that was the motivation to make pandas_market_calendars as a separate project.

Where are we now?

Over the last few years the projects have diverged, pandas_market_calendars now has different calendars than trading-calendars and each package has added new and different functionality in terms of methods and utility functions. While work to reconcile the calendars to make the best single source of truth that work is going on here quantopian/trading_calendars#152.

What are the benefits?

• In the philosophy of Python, there should be one way. Having a single source of market calendars benefits everyone. merging the projects will mean:
• One place to look to find what you need
• More eyes on the calendars increasing the ability to catch errors
• A larger community means more contributions of new calendars
• More eyes on one source, the better the quality

As a user of pandas_market_calendars, even if you don't care about the merger, you will immediately get access to all the trading calendars in trading_calendars.

What can I do?

• If you star or watch pandas_market_calendars, star or watch trading_calendars
• If you want to submit a PR for a new calendar, do it there. This is will pick it up automatically anyway

What is the process?

Three phases (each will match a git branch)

1. Expand pandas-market-calendars to include all the calendars in trading-calendars
2. Once all calendars are reconciled and identical between pandas_market_calendars and trading_calendars, the calendar implementation in pandas-market-calendars will be removed and direct to trading-calendars. See this for the work on making sure the calendars align: Data Differences Between trading_calendars and pandas_market_calendars quantopian/trading_calendars#152
3. Once all functionality in pandas-market-calendars is replicated in trading-calendars this project will be frozen. Method differences between trading_calendars and pandas_market_calendars quantopian/trading_calendars#151

Will this project die?

Until all the calendars and functionality is duplicated into trading-calendars this project will stay alive. Once trading-calendars is a complete replacement this project will go into a frozen state.

What if I depend on this project?

This project will always stay on GitHub and PyPi, just in a frozen state.

[rsheftel]

@jmccorriston
@richafrank
@gerrymanoim

[rsheftel]

Phase 1 to bring all the calendars in trading_calendar into pandas_market_calendar is on this branch https://github.com/rsheftel/pandas_market_calendars/tree/merge_phase1.0

At this point the code is complete, tests in progress.

[rsheftel]

Phase 1.0 is now complete in v1.6 here (8743bba) and pushed to PyPi

[rsheftel]

Phase 2.0 work has begun on this branch: https://github.com/rsheftel/pandas_market_calendars/tree/merge_phase2.0

[teathinker]

Now Quantopian's community platform is shutting down. How will that affect the other project and the merger?

[rsheftel]

@teathinker that is a good question. At this point I'll wait for the Quantopian guys that run the trading_calendars project to determine the future course of their project and then determine that path forward from there. The goals of the merger are still valid, this may just cause some disruption in timing. In the end it will be a community supported and completely open project regardless of the specifics of Quantopian business model.

[gerrymanoim]

Agreed with @rsheftel here. We're still sorting out a few things, but I think the goal of the merger is still very valid.

cc @richafrank

[rsheftel]

@jmccorriston
@richafrank
@gerrymanoim

Checking in on what you think about continuing the project to merge?

[gerrymanoim]

Hey - I'm still pro-merging the projects, but no longer really know the best way to proceed, since I lost some access when quantopian shut down. I can still review + merge PRs and release things, but can't merge my own PRs for fixes (I don't have any admin capabilities). However, I believe robinhood now owns the quantopian github org/main pypi accounts and I don't know what plans, if any, they have for trading_calendars in the future.

I believe that most of the data is the same between pandas_market_calendars and trading_calendars.

Happy to help however is useful, either in this repo or quantopian/trading_calendars or a new repo.

[rsheftel]

If the quantopian assets have been acquired by Robinhood, then my assumption is that eventually the quantopian Github repos will fall away. It does not appear that Robinhood has any existing open source Github projects so that does not seem to be the culture of their company (this isn't a critique of them, just statement of fact). As such I would say that doing more work in a repo you do not have control of is a risky choice.

My suggestion would be either:

1. clone and fork the quantopian/trading_calendars to your own so that you can properly control and manage it. And we can continue the plan on merging from there. This would work best if you have at least enough commit control to change the README to point everyone to the new fork so the quantopian one can go dormant

2. I put you guys as co-owner on this repo and you are free to use this directly to the be the source of the future merged work and you port the important parts of the trading_calendars over here. (as a bonus you can drop the requirements to have the 1min delayed open time as it isn't required for zipline)

@gerrymanoim let me know your thoughts

[gerrymanoim]

Hey @rsheftel - all good points. Completely agree with your high level thinking here. Unfortunately I don't have any real insight into what's going on at RH and what their future plans are.

1. I can still merge PRs (and you might still have approver access), I just cannot merge my own PRs. I'm happy to create a new fork under my account. Or I can fork it all under a new org that I invite you to. And then set up all the pypi/conda/actions/etc.

2. I think there was a lot of internal refactoring done in trading_calendars after pandas_market_calendars forked it that I'd like to keep, but I'm okay consolidating into here if you feel strongly. Either way I definitely want to drop the 1 minute delay and py2/pandas <1.

Let me know what you think.

[gerrymanoim]

I've started a fork and a todo list at gerrymanoim/exchange_calendars#1. Let me know if you have any comments/questions/thoughts/etc.

[rsheftel]

@gerrymanoim Why don't we do this:

1. You clean up trading_calendars like the plan you have on that open issue
2. Given that new refactoring we get back on looking how to do the merge

Do you still have access to change the README.md on the old trading_calendar repo? Can you put a notice on there that the repo is dead and moving to your new one?

[gerrymanoim]

Sounds good - I'll ping when I'm done.

RE the README: I can either PR it and have you approve or have you (or anyone else PR) and I'll approve.

[rsheftel]

@gerrymanoim I did the PR here: quantopian/trading_calendars#213 if you can approve

[gerrymanoim]

Approved and merged.

[gerrymanoim]

Okay - so I've published https://pypi.org/project/exchange-calendars/. There's a few other small things I'm thinking about doing (docs, etc) but the core functionality is there.

[rsheftel]

@gerrymanoim can you check that link, doesn't seem to work

[gerrymanoim]

Sorry - I pulled and republished that. Updated the link correctly now.

[kewlfft]

I am packaging for Arch Linux and getting confused between pandas_market_calendars, the original trading_calendars and the fork exchange_calendars. Should pandas_market_calendars have a dependency to the original or the fork?

Can you also repeat what you expect the target to be? I may directly switch to the new solution to avoid multiple transition steps.

Last question is there a calendar equivalent to EUREX in exchange_calendars or does EUREX need to be ported?

[rsheftel]

On the master branch, which is what people should use, the dependency is still on the original "trading_calendars" package. Over the next month I am going to make the changes required in pandas_market_calendars to move the dependency to the new fork "exchange_calendars".

Given that Quantopian is shut down and no one bought their assets, I would expect that "trading_calendars" will atrophy over time, so we are moving to sever any dependency on it. Again, targeting a month for that.

[kewlfft]

Thanks, my question on EUREX porting may be off topic but this is what prevents me from moving to exchange_calendars at the moment.

[rsheftel]

exchange_calendars fixes the bad opening time "feature" (bug) in the original "trading_calendars" package. I put a fix to back out the bug/feature in pandas_market_calendars. So all I need to do, when I get a moment, is to unwind that fix and it will work.

[rsheftel]

With the complete shutdown of Quantopian, the trading_calendars repo is now unmaintained. @gerrymanoim has forked the repo to https://github.com/gerrymanoim/exchange_calendars and that is where future development will take place. In addition he has corrected a lot of the quirks of the old package.

As of v2.0 of pandas_market_calendars (a798d6b) the mirror support for trading_calendars has been replaced with exchange_calendars.

At this point that completes the merger project. The plan of putting this package into a frozen mode will NOT happen. Instead it will continue to be actively maintained with the goal of keeping it as a means to access the calendars natively in this package or the vast number in the exchange_calendars package.

The goal of a single source of truth remains.

This is now closed.