Update to our customers on the recent Metrc API outages in California

You may have noticed over the past several weeks that Metrc submissions on Canix were taking longer than expected to sync with Metrc. This was a result of significant API downtime on Metrc. In translation, third-party integrators to Metrc -- such as Canix, we’re unable to process our normal volume of submissions. 

As part of our commitment to you, our customers, we want to ensure that you are informed especially when it relates to issues that impact your day-to-day. We also want to inform you of the steps we’re taking to mitigate this issue if it comes up in the future. 

What happened?

On Tuesday, March 4, 2021, Metrc sent out two bulletins 89 and 90. Both of these bulletins contained important information for Metrc’s third-party integrators, including Canix. 

These bulletins outlined a plan that Metrc was putting in place to remove several old API endpoints, which Canix uses to push and pull data from Metrc’s servers. 

In the process of updating of these endpoints, the Metrc API crashed affecting all third-party integrators. Beginning on March 12, Canix, along with all other Metrc third-party integrators began seeing a significant increase in failed submissions to Metrc. To put it plainly, a lot of data sent from Canix to Metrc wasn't transmitting due to this API failure. 

We have been in communications with Metrc and we aren't yet aware of the root cause of the issues but we know it was fixed around March 5th when submissions started succeeding again.

The graph below shows the number of times Canix retried Metrc Submissions between February 22nd and March 8th. After the fixes were implemented on March 4th, the retries decreased significantly. 

What is Canix doing?

The Canix Engineering team is well versed in working on APIs -- together we have a combined 60 years of experience building third-party integrations. 

Metrc Submissions

We have an incredibly talented team of Engineers who have built a number of safeguards into our system to ensure that your Metrc submissions are pushed through successfully. These include:

• Queued submissions. Even when Metrc is not down, Metrc can be slow to process transactions. All Canix Metrc Submissions are put in our automated Metrc submission queue you can see under Admin -> Metrc Submissions in Canix). Canix automatically retries submissions for you until the submissions are successfully processed by Metrc. If Metrc is slow to process, you can continue working until Metrc eventually processes your submission.
• Batching large submissions into no more than 500 items. We’ve found that attempting to submit more than 500 items at a time is more likely to result in failure. For example, if you are harvesting over 1,000 plants, Canix will batch that submission into 500 plants. The submission will all still appear under the same harvest when the job completes.
• If we're experiencing a backlog of submissions our system is built to re-try until they go through successfully. We've seen that in most instances for large plant/harvest submissions (1,000+ plants) it usually takes about 30 minutes to an hour for all data to be submitted.
• CSV Submissions. When the Metrc API goes down, the Metrc User Interface is often still up. Canix is currently building a feature that will allow users to download their Metrc submission as a CSV and upload it into the Metrc UI. Keep an eye out for a Product Update for this!

Communication

Aside from our technical prowess, transparency is at the core of everything we do. From in-product notifications to one of the most responsive Customer Support teams in the industry. In order to ensure that we’re maintaining that same level of communication, we’re releasing a new status page to which you can subscribe and get real-time updates on any incidents, scheduled maintenance, and system notices. 

Having a status page is a very common practice for software companies and is a great way to get information out quickly to as many customers as possible. If there were ever to be another outage like we’ve experienced over the past few weeks -- this tool will allow our team to send out an email, Tweet, and in-product notification about the issue as well as suggested workarounds. Giving you the power to take appropriate action and prevent any significant impact on your business.

What is Metrc doing?

• On February 17, Metrc’s technology team identified and corrected an error in a background job which, on initial investigation, was determined to be the potential cause of performance issues. 
• On February 18, Metrc significantly increased resources to servers for California while ongoing investigation to the issues ensued.
• On March 5, Metrc implemented the third tier of rate limiting in California. After the implementation of this rate limit, the number of calls that were failing in Metrc significantly decreased\
• On March 12, Metrc purged certain endpoints that were reportedly causing some issues in the API
• 
  

What can you do, as the customer?

While we have a very close relationship with Metrc, nothing is more important than the voice of the customer. If this downtime has materially affected your business, we suggest that you reach out directly to the BCC. We’ve provided a copy of some suggested language as well as the contacts to whom you should email your request.

Email to:

• BCCTrackandTrace@dca.ca.gov
• BCC@dca.ca.gov
• calcannabis@cdfa.ca.gov
• api-info@metrc.com

To whom it may concern,

In regards to the Metrc’s API reliability, we had a period of two weeks of downtime from dates 2/16/21 - 3/2/21 that caused significant issues with our compliance reporting. The California Business and Professions Code SEC. 51. Section 26068 mandates a functional API for licensees to report commercial cannabis activity, as required by regulation. We are writing to request that you resolve this issue by adding the appropriate safeguards. We are also requesting that you open up a more clear line of communication with your customers by releasing a status page that provides real-time updates on the health of the API and other Metrc functionality.