Subject:
Update on California Production Environment and Implementation of Phase 3 Rate Limits in California
Bulletin Number:
089
Distribution Date:
February 26, 2021
Effective Date:
March 5, 2021
Contact point:
api-info@metrc.com
Reason:
To provide third party integrators with an update regarding system performance in California, and notice of implementation of phase 3 Rate Limiting in the California instance
Hard copy of Bulletin:
Download PDF

Greetings Metrc® Integration Providers,
The Metrc team continues to field multiple reports on the performance and responsiveness of
Metrc in the California production system. As you are aware, these issues have been occurring
intermittently since February 17. This notice will provide you with what steps have already been
taken or are in progress to address these issues, and what additional steps are planned.
The additional steps include implementation of phase 3 rate limiting. The volume and pace of
calls to Metrc’s API has been a significant contributing factor to server load and applying the
rate limits in California that are already in place in other Metrc states will help alleviate that load.
Once those limits are in place in California, we will continue to evaluate these limits and
determine if modifications are necessary.


Most importantly, we want to acknowledge your patience, and that of your customers, as we
work to address system performance in California, and to apologize for integration disruptions.
While it is no doubt frustrating to have to report error messages or timeouts to the API support
team, we appreciate your continuing to do so in a timely manner, as such reports are a critical
aid to us in understanding and responding to system performance concerns.

Timeline

Initial performance concerns surfaced on Wednesday, February 17, and continued intermittently
to that Friday. Our mitigation measures (described below) appeared to be successful by Friday
afternoon, though scattered reports of performance concerns came in Saturday as well. On
Monday, February 22, performance again degraded, and these issues have recurred
intermittently this week.

Mitigation Measures

• On February 17, Metrc’s Technology team identified and corrected an error in a background
job which, on initial investigation, was determined to be the potential cause to performance
issues.
• On February 18, Metrc significantly increased resources on our servers for California while
ongoing investigation to the issues ensued.
• Throughout this process, we have reached out to integrators who were generating high
volumes of API traffic to discuss their business needs and how they might be met without
impacting overall system performance. The integrators we have reached out to have been
uniformly responsive and constructive both in sharing information and in modulating their
traffic to Metrc, and we thank all involved in that process.

Ongoing Efforts
Metrc continues to take steps to improve this situation:
• Further server upgrades are in the works. We are anticipating those changes will require
a brief production outage; if so, we will schedule that outage so as to minimize the
impact on licensees and integrators, and we will provide notice to Metrc online users via
a banner message within the application.
• We are investigating other potential architectural changes and will provide you with
notice of such changes if they are not expected to be transparent to you.
• We also will be implementing Rate Limiting Phase 3 as another measure to improve the
performance situation.

How You Can Help

In addition to the work being done on the Metrc end, we wanted to also outline some items that
would be helpful for the integrators to consider.
1. Apply the best practices within the Metrc API Documentation for maintaining inventory
in sync and requesting multiple days of data.
2. Consider any large jobs that require pulling data out of Metrc to be run in off business
hours (e.g., “between 9pm and 7am”)
3. Apply a “wait time” between API requests, especially if you notice slower response
times.

Rate Limiting Update

As we noted in multiple previous bulletins, starting with bulletin 70 issued on July 14, 2020, we
are phasing in rate limits for third-party integrators. Our goal is to improve system performance
for all users by limiting the number of API calls a third-party integrator can make in a given
amount of time to ensure no single user or company can overwhelm the system.
Rate limiting is a best practice used by software platforms worldwide, such as Twitter, LinkedIn,
Slack, Quick Books, and Google. Metrc’s rate limiting approach is intentionally designed to
gradually phase in so integrators have ample time to adjust their systems as necessary.
We would like to announce that for the California instance of Metrc, the third rate limit will be
implemented, on March 5th, 2021. This date was selected to strike a balance between
providing advance notice to integrators and alleviating the issues noted in this bulletin. In
addition to this notice, Metrc will reach out to those integrators we expect to be most impacted
by the phase 3 rate limits.


Phase 3 Rate Limits


• Go-Live Date: January 21, 2021 (excluding California)/ March 5, 2021 (California)
• Rate Limits (all implemented simultaneously):
o 50 GET calls per second per facility,
o 150 GET calls per second per integrator key,
o 10 concurrent GET calls per facility, and

o 30 concurrent GET calls per integrator
• Total Allowable Daily Calls per Integrator:
o 12.96 million per day per integrator key
• Non-Affected Calls: Rate limits do not apply to PUT, POST, or DELETE calls
See our API Resources page for further information on Metrc rate limiting, including technical
information: https://www.metrc.com/api-resources
We appreciate your understanding and support. Please contact api-info@metrc.com with any
questions or concerns.

Where are You Located?

Do you have over 15 employees?

  Yes      No

Are you operating in multiple states?

  Yes      No

Next
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.