tag:status.hyra.io,2005:/historyHyra Status - Incident History2024-03-28T06:50:35+00:00Hyratag:status.hyra.io,2005:Incident/202849062024-03-17T17:35:21+00:002024-03-17T17:35:21+00:00Issues accessing Hyra for some users<p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>17:35</var> GMT</small><br><strong>Resolved</strong> - A permanent fix has been applied and the issue has now been resolved.<br /><br />We apologise to our customers for the downtime of approximately 8 minutes experienced with us today. Today's downtime was caused by an expired SSL certificate. We have put safeguards in place to prevent the expiry of these certificates in the future to mitigate the possibility of similar incidents in the future. Thanks for your understanding and for being a Hyra customer.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>17:20</var> GMT</small><br><strong>Monitoring</strong> - A temporary fix has been implemented and we are monitoring the results</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>17:18</var> GMT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>17:09</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.hyra.io,2005:Incident/200661342024-02-24T20:48:33+00:002024-02-24T20:48:33+00:00Roblox Outage<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>20:48</var> GMT</small><br><strong>Resolved</strong> - We have confirmed the API is now stable and restored.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>17:40</var> GMT</small><br><strong>Identified</strong> - We have identified the issue to be related to a Roblox issue on the Groups API. We are now upping this to a partial outage. This issue is outside of our control, but we will continue monitoring systems to ensure stability is resumed as soon as possible.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>17:36</var> GMT</small><br><strong>Investigating</strong> - We're investigating a report that Activity is tracking intermittently on Hyra. We will post updates as we have them.</p>tag:status.hyra.io,2005:Incident/199998752024-02-17T16:00:56+00:002024-02-17T16:00:56+00:00Routine Maintenance<p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>16:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>15:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>13:24</var> GMT</small><br><strong>Scheduled</strong> - To improve our products and services and to make sure everything continues to run smoothly, we conduct routine maintenance.<br /><br />During this time, we will be performing software upgrades and patches to our central servers, so services may be unreachable whilst we carry this out. We apologise for the inconvenience and aim to cause as little disruption as we can.</p>tag:status.hyra.io,2005:Incident/199102222024-02-06T08:00:02+00:002024-02-06T08:00:02+00:00Database Maintenance<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>08:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>07:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>07:52</var> GMT</small><br><strong>Scheduled</strong> - We are performing a hardware upgrade on one of our database nodes. While we carry out this maintenance, you'll be unable to access or log into Hyra. We're sorry for the inconvenience caused and anticipate maintenance to last no longer than 15 minutes.</p>tag:status.hyra.io,2005:Incident/199186752024-02-06T05:42:00+00:002024-02-06T05:42:00+00:00Issues accessing Hyra<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>05:42</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>05:40</var> GMT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>05:39</var> GMT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>05:35</var> GMT</small><br><strong>Investigating</strong> - We're aware that some customers are unable to access Hyra and are looking into the root cause.</p>tag:status.hyra.io,2005:Incident/198356682024-02-01T11:00:36+00:002024-02-01T11:00:36+00:00Platform Maintenance<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>11:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>08:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>07:19</var> GMT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance of our APIs to improve reliability and performance from 1st February 2024 at 08:00 GMT until 11:00 GMT. During this time, services may be inaccessible to some or all users. We aim to have a downtime of less than ~5 minutes in this time window.<br /><br />We apologise for any inconvenience this maintenance causes to our customers. This type of maintenance is essential for improving our platform in the longer term. We greatly appreciate your understanding and support.</p>tag:status.hyra.io,2005:Incident/198246872024-01-26T07:10:09+00:002024-01-26T07:10:09+00:00Routine Maintenance<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>07:10</var> GMT</small><br><strong>Completed</strong> - We've completed the scheduled maintenance. We're sorry that there was more downtime than expected (approx. 20 minutes). Everything is now operational. Our team will continue to monitor services as always to ensure correct operation.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>07:08</var> GMT</small><br><strong>Update</strong> - We're performing final checks to confirm everything is operational. Sit tight!</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>07:06</var> GMT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>06:42</var> GMT</small><br><strong>In progress</strong> - We have started out maintenance ahead of schedule. Updates will be provided as required.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>02:46</var> GMT</small><br><strong>Scheduled</strong> - We will be undergoing routine maintenance for upgrades and security patches. Services may be slow or unavailable during this time, though we will try our best to avoid this where possible.</p>tag:status.hyra.io,2005:Incident/196351172024-01-13T21:01:07+00:002024-01-13T21:01:07+00:00Subnet Maintenance<p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>21:01</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>20:01</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>07:41</var> GMT</small><br><strong>Scheduled</strong> - We will be performing maintenance to our internal network that may affect production services.</p>tag:status.hyra.io,2005:Incident/196345772024-01-07T06:34:46+00:002024-01-07T06:49:04+00:00Some users unable to access Hyra<p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>06:34</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>06:32</var> GMT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>06:15</var> GMT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>06:09</var> GMT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>06:01</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.hyra.io,2005:Incident/195438722023-12-27T18:31:12+00:002023-12-27T18:31:12+00:00Ranking Maintenance<p><small>Dec <var data-var='date'>27</var>, <var data-var='time'>18:31</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>27</var>, <var data-var='time'>17:01</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>27</var>, <var data-var='time'>16:20</var> GMT</small><br><strong>Scheduled</strong> - We are migrating our Ranking Services to another region of our cloud infrastructure. We will endeavour to have 0 downtime during this process. Please be advised that the IP addresses used for ranking will change. We will post further information once it is available.</p>tag:status.hyra.io,2005:Incident/194701332023-12-18T14:20:59+00:002023-12-18T14:20:59+00:00Database Maintenance<p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>14:20</var> GMT</small><br><strong>Completed</strong> - The maintenance has now been completed. We apologise to customers that we ran over schedule, although most customer resources were restored on time.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>14:15</var> GMT</small><br><strong>Update</strong> - We are continuing to verify the maintenance items.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>14:06</var> GMT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>14:00</var> GMT</small><br><strong>Update</strong> - Maintenance is still in progress as we work to restore services.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>13:48</var> GMT</small><br><strong>Update</strong> - Maintenance is now well underway and we are working to restore services.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>13:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>12:47</var> GMT</small><br><strong>Scheduled</strong> - We are performing database security upgrades during this time.<br /><br />We apologise for the short notice and any inconvenience caused. Unfortunately, some downtime is anticipated during this maintenance. We will aim to keep service disruption to a minimum.</p>tag:status.hyra.io,2005:Incident/194069822023-12-12T16:00:34+00:002023-12-12T16:00:34+00:00Network Maintenance<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>16:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>14:01</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>01:54</var> GMT</small><br><strong>Scheduled</strong> - We will be undergoing essential network maintenance during this time.<br /><br />We aim for there to be no service downtime to customer services, however there may be increased latency as we perform migration and upgrades.<br /><br />These upgrades are essential for security, maintainability and improvements.<br /><br />Hyra apologises for the inconvenience and appreciates your understanding.</p>tag:status.hyra.io,2005:Incident/191310052023-11-15T21:29:57+00:002023-11-15T21:29:57+00:00Partial outage<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>21:29</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>21:23</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.hyra.io,2005:Incident/189931022023-11-01T20:25:28+00:002023-11-01T23:21:40+00:00Increased API Errors<p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>20:25</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>20:16</var> GMT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>20:15</var> GMT</small><br><strong>Monitoring</strong> - The issue has been identified as an issue with our GCP edge router. We are working with our infra provider to resolve issues.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>20:09</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating an increased amount of API Errors.</p>tag:status.hyra.io,2005:Incident/189714192023-10-30T20:55:47+00:002023-10-30T21:22:11+00:00Services unavailable<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>20:55</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>20:55</var> GMT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>20:24</var> GMT</small><br><strong>Update</strong> - An incident can be tracked on Cloudflare Status now: https://www.cloudflarestatus.com/incidents/l6x2h1zp69bc</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>20:11</var> GMT</small><br><strong>Update</strong> - We are continuing to work on a fix.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>20:03</var> GMT</small><br><strong>Identified</strong> - We have identified the issue as an upstream provider, Cloudflare. We are looking into solutions.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>20:00</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.hyra.io,2005:Incident/187224412023-10-08T09:29:08+01:002023-10-08T09:29:08+01:00Issue with assignment marking<p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>09:29</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>09:17</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>09:11</var> BST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>08:58</var> BST</small><br><strong>Investigating</strong> - We're currently investigating reports that assignments have not been marked for Sunday - Sunday week ending 8th. We will provide updates as required.</p>tag:status.hyra.io,2005:Incident/186612562023-10-02T10:57:08+01:002023-10-02T10:57:08+01:00Avatar Service Maintenance<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>10:57</var> BST</small><br><strong>Completed</strong> - Maintenance has now been completed and the endpoint has been successfully migrated.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>07:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>00:44</var> BST</small><br><strong>Scheduled</strong> - We’re making optimisations to our Avatar Service. During this time, avatars may fail to load. We’ll try to keep disruption to a minimum, but due to the nature of how this new service will be deployed, some downtime is anticipated. <br /><br />This is a cosmetic service.</p>tag:status.hyra.io,2005:Incident/186386292023-09-29T08:00:08+01:002023-09-29T08:00:08+01:00Activity service maintenance<p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>08:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>07:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>04:21</var> BST</small><br><strong>Scheduled</strong> - We will be conducting maintenance to our activity service during this time. This maintenance is part of a wider improvement to Hyra. We will be migrating this service to different infrastructure.<br /><br />While no downtime is expected, and we will aim to reduce downtime as much as possible, it is a possibility we cannot rule out. If downtime should occur, activity will not be tracked for players in affected servers.</p>tag:status.hyra.io,2005:Incident/184118262023-09-06T20:33:48+01:002023-09-06T22:34:02+01:00Issues with synchronous tasks<p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>20:33</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>20:01</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>19:31</var> BST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>19:28</var> BST</small><br><strong>Investigating</strong> - We're aware and investigating issues regarding tasks like member syncing and assignments and are looking into the problem.</p>tag:status.hyra.io,2005:Incident/182731372023-08-26T18:40:01+01:002023-08-26T18:40:01+01:00Maintenance to workers<p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>18:40</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>16:41</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>16:39</var> BST</small><br><strong>Scheduled</strong> - We're completing maintenance on our backend service workers, responsible for behind the scenes tasks like assignments, sessions, and syncing workspaces. This maintenance is part of Hyra's ongoing effort to improve stability to the platform.<br /><br />We apologise about the short notice for this maintenance - this maintenance has been pushed forward as part of preventative maintenance.</p>tag:status.hyra.io,2005:Incident/180674662023-08-06T17:54:16+01:002023-08-06T17:54:16+01:00Assignments not marked<p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>17:54</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>17:40</var> BST</small><br><strong>Update</strong> - We have finalised our fixes and are monitoring all results</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>17:09</var> BST</small><br><strong>Update</strong> - We are continuing to make improvements to the pipeline.</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>16:27</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented, and we are now testing and monitoring the results. Assignment history may appear and then disappear during this testing phase.</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>16:02</var> BST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented. The timeline for implementing the fix is no longer than 3 hours.</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>15:53</var> BST</small><br><strong>Update</strong> - We are continuing to investigate the root cause of the issue.</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>15:42</var> BST</small><br><strong>Investigating</strong> - We're currently investigating reports of assignments not being marked correctly. We'll post continuous updates as we have them.</p>tag:status.hyra.io,2005:Incident/179120242023-07-20T14:39:06+01:002023-07-20T14:39:06+01:00Issues with registering accounts<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>14:39</var> BST</small><br><strong>Resolved</strong> - The issue has been confirmed resolved by the frontline team and users can now sign up. We apologise about the inconvenience caused by this and have made changes to prevent this from happening again in the future.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>14:36</var> BST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>14:36</var> BST</small><br><strong>Investigating</strong> - We're currently aware and investigating issues with signing up for Hyra.</p>tag:status.hyra.io,2005:Incident/176997882023-06-29T18:00:24+01:002023-06-29T18:00:24+01:00Engineering works<p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>18:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>15:01</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>15:35</var> BST</small><br><strong>Scheduled</strong> - We are conducting engineering maintenance to our critical infrastructure to improve reliability. During this time, services may be slower or entirely inaccessible. We apologise for the inconvenience.</p>tag:status.hyra.io,2005:Incident/158963682023-01-23T06:00:40+00:002023-01-23T06:00:40+00:00Engineering Works<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>06:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>03:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>01:21</var> GMT</small><br><strong>Scheduled</strong> - Essential engineering works to our authentication service</p>tag:status.hyra.io,2005:Incident/158963662023-01-22T06:02:53+00:002023-01-22T06:02:53+00:00Engineering Works<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>06:02</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>03:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>01:21</var> GMT</small><br><strong>Scheduled</strong> - Essential engineering works to our authentication service</p>