Notes
We started off with an update on the current status of outages and degraded service. On 8-21, we made a change to our virtual private cloud settings that resulted in the entire API being non-functional. We weren’t aware of this until the morning of 8-22. It was a quick fix once found, but the API was out for more than twelve hours. We’ve made changes to our alerting and notification policies to avoid this happening again.
We also have an ongoing degradation of services around missing data. Several endpoints are currently running on cached data, as new data is not available upstream:
- Award billing schedules
- Award customer contract lines
- Sales items
- Tax codes
In addition, we’re still seeing some incorrect results from the FDM validation endpoint due to missing data. Usually this looks like a combo that should be valid being marked invalid. We’re working with other teams to address this.
Now, for what we did: we made a new endpoint for ISD lines. We realized that we had the wrong data model for ISDs and hadn’t accounted for ISDs that contain multiple lines. The existing endpoint has been left as is, but is deprecated. The new endpoint includes all the ISD lines and includes the fields from the parent ISD record.
We fixed a rare bug that was causing server errors when datetime parameters were not properly formatted. You should see a 400 error now, indicating you need to fix your request before a response can be issued. We also made some internal improvements to logging that will help us troubleshoot problems with the FDM validator. If you’re getting a server error with a large batch of FDM combos, let us know and we’ll find the one(s) that are the source of the problem.
We made some changes to our deployment processes to run deployments more frequently. We hope this will reduce the burden of running a deploy and allow new features to come out sooner. It’s still experimental, so we’ll evaluate it at the end of the next sprint. We also started running our ETL process via Gitlab, which gives us some new options for triggering it automatically in response to other events.
We’re aware of a bug that’s causing the example responses in the Developer Portal to not appear on the page. The examples are still in the specification, but the software behind the Portal isn’t displaying them correctly. We’re working on a fix to this.
API Changes
- New isdLines endpoint