Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 747875 reports (Page 12784 of 14958)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.285668934s FAIL (2025-08-12 09:41:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.285690215s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.......X........X...........] duration: 1.819927024s FAIL (2025-08-12 09:41:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.819947964s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.321651611s FAIL (2025-08-12 09:40:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.321674526s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.545183077s FAIL (2025-08-12 09:40:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.545202515s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X......X......................X....X....................X] duration: 5.827742992s FAIL (2025-08-12 09:40:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 0fc96fde5806e1e3de92faea4b72f1ae57abc377489f6cedba17470d4f5cc0fc != before 7232133e1c2ebdac11d279c7897a837825c5db6123d391365771f0726c6bf27e ======LOGS====== Aug 12 09:40:33.606658726 (88h14m28.037738877s) [upload=91928]: Starting upload Aug 12 09:40:33.606785179 (88h14m28.03786532s) [upload=91928]: Done starting upload Aug 12 09:40:33.606937930 (88h14m28.038018101s) [upload=91928]: Uploading segments... Aug 12 09:40:33.607018985 (88h14m28.038099116s) [upload=91928]: Got next segment. Inline: false Aug 12 09:40:33.607060766 (88h14m28.038140897s) [upload=91928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:40:33.607066957 (88h14m28.038147109s) [upload=91928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 09:40:34.092146890 (88h14m28.523227011s) [seg_pos={0 0}, upload=91928]: Begin upload segment... Aug 12 09:40:34.092201394 (88h14m28.523281566s) [seg_pos={0 0}, upload=91928]: Done begin upload segment. Aug 12 09:40:34.092234227 (88h14m28.523314358s) [noise=true , address=104.36.82.62:4416 , piece_id=3ZNZMSDHIMIVMP25RLQTQI4L5LY3HMU3BGJQ5INAFXWA3NWGV2HQ, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.092250709 (88h14m28.52333088s) [noise=true , address=66.23.193.175:4563 , piece_id=OFSROHILQI7LJOEKVX3UL233THLP5B2IT5BIIK3FSBXPEB7OMWFA, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.092625896 (88h14m28.523706017s) [noise=true , address=104.36.82.61:4402 , piece_id=DJE46LL2UNHIFWOYULJUDMABJQ7J5L7SYH7GQJ6C4RGV4NZMMXYA, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.092715318 (88h14m28.523795449s) [noise=true , address=64.49.13.140:4513 , piece_id=ZM3T7JFFPY6XSRJWFHBXJUFSY76LGD2YKPZQHHXZUTZEYG3V4QLQ, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.092883750 (88h14m28.523963881s) [noise=true , address=77.90.31.15:4714 , piece_id=VWUZ4JXRHP5IIJRKF7ROQVOQYEK3MYIXLYUZUF4SXNGT735PI7MQ, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.092994432 (88h14m28.524074563s) [noise=true , address=104.36.82.16:4299 , piece_id=W2XFF675K4462EL6JQDRFVBYT254PWQ27V62KAOGQITOWV5QRMYQ, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.093125874 (88h14m28.524206005s) [noise=true , address=79.127.201.215:4347 , piece_id=6SSDS5ENEBR2SMLPH3NU4PATJ4MG2JGLBCH6JPUZU6UE33S73Z5A, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.093268146 (88h14m28.524348267s) [noise=true , address=104.36.82.60:4349 , piece_id=LN4ALALZSPNHINMVTXOTFGV3WPWRY52GZ3EBFO75PKGQHU7UJYQQ, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.093331827 (88h14m28.524411958s) [noise=true , address=104.36.82.23:4468 , piece_id=SJOJD7G72A4AINLYJ323DUSYP3B2G7CPOWUVKNCSUJYTJ4VS6FSQ, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.093486634 (88h14m28.524566765s) [noise=true , address=104.36.82.34:4569 , piece_id=NE2D2STL745MKBOMINWN565HVOYIAGIEZXGLPNQVIUX3TNI3AGLA, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.093697537 (88h14m28.524777668s) [noise=true , address=66.23.193.175:4561 , piece_id=6Q5TREIQ6Q6ESXGODB5X6FNQ4J3BQFE62BTURRMGS74MFWEKMESQ, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.093932497 (88h14m28.525012628s) [noise=true , address=79.127.220.101:4070 , piece_id=JJICX25SOA7KBITCMJEKHY3U3SQRD6UINJCLFP4IFLCQUUABZXDA, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.094170552 (88h14m28.525250683s) [noise=true , address=104.36.82.44:4900 , piece_id=BLEA4M6WAWWGBRS3L2GWQ3HDYX2Z7O6US6UKRRKS6LNFHV22LBMA, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.094525271 (88h14m28.525605402s) [noise=true , address=104.36.82.38:4703 , piece_id=ZKEKJEOLWFJ7BOAFHQZV5FMZWV55SD3DCYWPXHC6L4GTB3ITAFIA, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.094885360 (88h14m28.525965491s) [noise=true , address=64.49.13.169:4716 , piece_id=GUURL7MO7PNRNEUMDSEJY5FFFUSVTJWZ2SGJGEZESFJJZCG3JPOQ, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.095084912 (88h14m28.526165033s) [noise=true , address=104.36.82.37:4678 , piece_id=TADG5GXRS22D5OGC7ABOYUG2N3APPAFHIVY63D5L6XYGRNHCRT5Q, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.095275196 (88h14m28.526355327s) [noise=true , address=77.90.31.17:4802 , piece_id=YPHII43XXBNGDN2VTPAN3B7HSU3YYPA7FTASCO7XWY23CAXOGDHQ, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:34.095472042 (88h14m28.526552173s) [noise=true , address=77.90.31.19:4851 , piece_id=3JC4JUTW3ONK5PSGPULTEEBBMLSHYDBHJH3WHCYULAPZFV7DECPA, seg_pos={0 0}, upload=91928]: Uploading piece... Aug 12 09:40:35.240468370 (88h14m29.671548502s) [upload=91928]: Next returned nil segment Aug 12 09:40:35.240485464 (88h14m29.671565595s) [upload=91928]: Waiting for error group managing segments... Aug 12 09:40:35.528522331 (88h14m29.959602462s) [noise=true , address=64.49.13.169:4716 , piece_id=GUURL7MO7PNRNEUMDSEJY5FFFUSVTJWZ2SGJGEZESFJJZCG3JPOQ, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:35.556931408 (88h14m29.988011529s) [noise=true , address=66.23.193.175:4563 , piece_id=OFSROHILQI7LJOEKVX3UL233THLP5B2IT5BIIK3FSBXPEB7OMWFA, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:35.564981912 (88h14m29.996062043s) [noise=true , address=66.23.193.175:4561 , piece_id=6Q5TREIQ6Q6ESXGODB5X6FNQ4J3BQFE62BTURRMGS74MFWEKMESQ, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:35.582428493 (88h14m30.013508624s) [noise=true , address=64.49.13.140:4513 , piece_id=ZM3T7JFFPY6XSRJWFHBXJUFSY76LGD2YKPZQHHXZUTZEYG3V4QLQ, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:35.932112053 (88h14m30.363192184s) [noise=true , address=79.127.220.101:4070 , piece_id=JJICX25SOA7KBITCMJEKHY3U3SQRD6UINJCLFP4IFLCQUUABZXDA, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:36.011282054 (88h14m30.442362185s) [noise=true , address=77.90.31.19:4851 , piece_id=3JC4JUTW3ONK5PSGPULTEEBBMLSHYDBHJH3WHCYULAPZFV7DECPA, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:36.029115326 (88h14m30.460195457s) [noise=true , address=104.36.82.23:4468 , piece_id=SJOJD7G72A4AINLYJ323DUSYP3B2G7CPOWUVKNCSUJYTJ4VS6FSQ, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:36.031812817 (88h14m30.462892948s) [noise=true , address=104.36.82.38:4703 , piece_id=ZKEKJEOLWFJ7BOAFHQZV5FMZWV55SD3DCYWPXHC6L4GTB3ITAFIA, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:36.045286577 (88h14m30.476366708s) [noise=true , address=104.36.82.34:4569 , piece_id=NE2D2STL745MKBOMINWN565HVOYIAGIEZXGLPNQVIUX3TNI3AGLA, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:36.046141442 (88h14m30.477221563s) [noise=true , address=104.36.82.16:4299 , piece_id=W2XFF675K4462EL6JQDRFVBYT254PWQ27V62KAOGQITOWV5QRMYQ, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:36.054419882 (88h14m30.485500014s) [noise=true , address=104.36.82.61:4402 , piece_id=DJE46LL2UNHIFWOYULJUDMABJQ7J5L7SYH7GQJ6C4RGV4NZMMXYA, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:36.066340240 (88h14m30.497420361s) [noise=true , address=79.127.201.215:4347 , piece_id=6SSDS5ENEBR2SMLPH3NU4PATJ4MG2JGLBCH6JPUZU6UE33S73Z5A, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:36.074943862 (88h14m30.506023993s) [noise=true , address=77.90.31.17:4802 , piece_id=YPHII43XXBNGDN2VTPAN3B7HSU3YYPA7FTASCO7XWY23CAXOGDHQ, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:36.135618764 (88h14m30.566698895s) [noise=true , address=104.36.82.60:4349 , piece_id=LN4ALALZSPNHINMVTXOTFGV3WPWRY52GZ3EBFO75PKGQHU7UJYQQ, seg_pos={0 0}, upload=91928]: Done uploading piece. err: <nil> Aug 12 09:40:36.135648783 (88h14m30.566728914s) [seg_pos={0 0}, upload=91928]: Segment reached optimal threshold of 14 pieces. Aug 12 09:40:36.135792988 (88h14m30.566873119s) [noise=true , address=104.36.82.37:4678 , piece_id=TADG5GXRS22D5OGC7ABOYUG2N3APPAFHIVY63D5L6XYGRNHCRT5Q, seg_pos={0 0}, upload=91928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Eqo4D6dYxJBJXN3Ww2FYoL2cFudeBSyJQdbKqnuMAS6mZ4En6): protocol: expected piece hash; context canceled; EOF Aug 12 09:40:36.135833976 (88h14m30.566914107s) [noise=true , address=104.36.82.44:4900 , piece_id=BLEA4M6WAWWGBRS3L2GWQ3HDYX2Z7O6US6UKRRKS6LNFHV22LBMA, seg_pos={0 0}, upload=91928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R8K65zsCspu7W2oAnvtpz7zEsQFkDG1e5BAp6faQYMQrkh4oQ): protocol: expected piece hash; context canceled; EOF Aug 12 09:40:36.135867221 (88h14m30.566947352s) [noise=true , address=77.90.31.15:4714 , piece_id=VWUZ4JXRHP5IIJRKF7ROQVOQYEK3MYIXLYUZUF4SXNGT735PI7MQ, seg_pos={0 0}, upload=91928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Aug 12 09:40:36.135913058 (88h14m30.566993189s) [noise=true , address=104.36.82.62:4416 , piece_id=3ZNZMSDHIMIVMP25RLQTQI4L5LY3HMU3BGJQ5INAFXWA3NWGV2HQ, seg_pos={0 0}, upload=91928]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12syasUPsyYf4UstQ6cFAr6qamze3vCyAm9dnobk5e83w4bTz69): protocol: expected piece hash; context canceled; EOF Aug 12 09:40:36.135924670 (88h14m30.567004801s) [seg_pos={0 0}, upload=91928]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 12 09:40:36.135980517 (88h14m30.567060638s) [upload=91928]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 09:40:36.135984755 (88h14m30.567064886s) [upload=91928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:40:36.422758204 (88h14m30.853838345s) [upload=91928]: Done uploading segments... Aug 12 09:40:36.422780928 (88h14m30.853861069s) [upload=91928]: Upload finished. err: <nil> Duration: 5.827768892s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.258050359s FAIL (2025-08-12 09:40:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.258074455s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [................................X.......X........X..........] duration: 1.81359046s FAIL (2025-08-12 09:40:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.813618873s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X......X......................X....X....................] duration: 5.72418635s FAIL (2025-08-12 09:39:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:23:08.516266754 (87h57m2.947346885s) [upload=91621]: Starting upload Aug 12 09:23:08.516319585 (87h57m2.947399716s) [upload=91621]: Done starting upload Aug 12 09:23:08.516447219 (87h57m2.947527371s) [upload=91621]: Uploading segments... Aug 12 09:23:08.516526641 (87h57m2.947606772s) [upload=91621]: Got next segment. Inline: false Aug 12 09:23:08.516564794 (87h57m2.947644925s) [upload=91621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:23:08.516570595 (87h57m2.947650736s) [upload=91621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 09:23:08.760094669 (87h57m3.1911748s) [upload=91621]: Next segment err: metaclient: Too Many Requests Aug 12 09:23:08.760114537 (87h57m3.191194668s) [upload=91621]: Done uploading segments... Aug 12 09:23:08.760127261 (87h57m3.191207392s) [upload=91621]: Upload finished. err: metaclient: Too Many Requests Duration: 5.724243429s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.340138944s FAIL (2025-08-12 09:39:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.34015742s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.118427093s FAIL (2025-08-12 09:39:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.118476278s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.......X........X.........] duration: 1.866004827s FAIL (2025-08-12 09:39:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.866030925s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.790508115s FAIL (2025-08-12 09:39:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.790535707s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X......X......................X....X...................] duration: 5.827712985s FAIL (2025-08-12 09:38:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:23:08.516266754 (87h57m2.947346885s) [upload=91621]: Starting upload Aug 12 09:23:08.516319585 (87h57m2.947399716s) [upload=91621]: Done starting upload Aug 12 09:23:08.516447219 (87h57m2.947527371s) [upload=91621]: Uploading segments... Aug 12 09:23:08.516526641 (87h57m2.947606772s) [upload=91621]: Got next segment. Inline: false Aug 12 09:23:08.516564794 (87h57m2.947644925s) [upload=91621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:23:08.516570595 (87h57m2.947650736s) [upload=91621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 09:23:08.760094669 (87h57m3.1911748s) [upload=91621]: Next segment err: metaclient: Too Many Requests Aug 12 09:23:08.760114537 (87h57m3.191194668s) [upload=91621]: Done uploading segments... Aug 12 09:23:08.760127261 (87h57m3.191207392s) [upload=91621]: Upload finished. err: metaclient: Too Many Requests Duration: 5.827731571s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.854645958s FAIL (2025-08-12 09:38:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.854665546s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.62211721s FAIL (2025-08-12 09:38:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.622140967s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.......X........X........] duration: 1.839410433s FAIL (2025-08-12 09:38:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.839436183s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.133574629s FAIL (2025-08-12 09:38:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.133595771s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.297493971s FAIL (2025-08-12 09:37:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.297517166s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X......X......................X....X..................] duration: 6.420991727s FAIL (2025-08-12 09:37:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:23:08.516266754 (87h57m2.947346885s) [upload=91621]: Starting upload Aug 12 09:23:08.516319585 (87h57m2.947399716s) [upload=91621]: Done starting upload Aug 12 09:23:08.516447219 (87h57m2.947527371s) [upload=91621]: Uploading segments... Aug 12 09:23:08.516526641 (87h57m2.947606772s) [upload=91621]: Got next segment. Inline: false Aug 12 09:23:08.516564794 (87h57m2.947644925s) [upload=91621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:23:08.516570595 (87h57m2.947650736s) [upload=91621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 09:23:08.760094669 (87h57m3.1911748s) [upload=91621]: Next segment err: metaclient: Too Many Requests Aug 12 09:23:08.760114537 (87h57m3.191194668s) [upload=91621]: Done uploading segments... Aug 12 09:23:08.760127261 (87h57m3.191207392s) [upload=91621]: Upload finished. err: metaclient: Too Many Requests Duration: 6.421021516s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.408309382s FAIL (2025-08-12 09:37:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.408331053s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X......X......................X....X.................] duration: 5.663637935s FAIL (2025-08-12 09:37:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:23:08.516266754 (87h57m2.947346885s) [upload=91621]: Starting upload Aug 12 09:23:08.516319585 (87h57m2.947399716s) [upload=91621]: Done starting upload Aug 12 09:23:08.516447219 (87h57m2.947527371s) [upload=91621]: Uploading segments... Aug 12 09:23:08.516526641 (87h57m2.947606772s) [upload=91621]: Got next segment. Inline: false Aug 12 09:23:08.516564794 (87h57m2.947644925s) [upload=91621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:23:08.516570595 (87h57m2.947650736s) [upload=91621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 09:23:08.760094669 (87h57m3.1911748s) [upload=91621]: Next segment err: metaclient: Too Many Requests Aug 12 09:23:08.760114537 (87h57m3.191194668s) [upload=91621]: Done uploading segments... Aug 12 09:23:08.760127261 (87h57m3.191207392s) [upload=91621]: Upload finished. err: metaclient: Too Many Requests Duration: 5.663665137s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.......X........X.......] duration: 1.800730405s FAIL (2025-08-12 09:37:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.800758528s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X......X......................X....X................] duration: 5.631423503s FAIL (2025-08-12 09:36:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:23:08.516266754 (87h57m2.947346885s) [upload=91621]: Starting upload Aug 12 09:23:08.516319585 (87h57m2.947399716s) [upload=91621]: Done starting upload Aug 12 09:23:08.516447219 (87h57m2.947527371s) [upload=91621]: Uploading segments... Aug 12 09:23:08.516526641 (87h57m2.947606772s) [upload=91621]: Got next segment. Inline: false Aug 12 09:23:08.516564794 (87h57m2.947644925s) [upload=91621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:23:08.516570595 (87h57m2.947650736s) [upload=91621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 09:23:08.760094669 (87h57m3.1911748s) [upload=91621]: Next segment err: metaclient: Too Many Requests Aug 12 09:23:08.760114537 (87h57m3.191194668s) [upload=91621]: Done uploading segments... Aug 12 09:23:08.760127261 (87h57m3.191207392s) [upload=91621]: Upload finished. err: metaclient: Too Many Requests Duration: 5.631440986s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.279240515s FAIL (2025-08-12 09:36:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.279256154s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.133639683s FAIL (2025-08-12 09:36:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.133654381s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X........X......X......................X....X...............] duration: 5.976153304s FAIL (2025-08-12 09:36:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:23:08.516266754 (87h57m2.947346885s) [upload=91621]: Starting upload Aug 12 09:23:08.516319585 (87h57m2.947399716s) [upload=91621]: Done starting upload Aug 12 09:23:08.516447219 (87h57m2.947527371s) [upload=91621]: Uploading segments... Aug 12 09:23:08.516526641 (87h57m2.947606772s) [upload=91621]: Got next segment. Inline: false Aug 12 09:23:08.516564794 (87h57m2.947644925s) [upload=91621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:23:08.516570595 (87h57m2.947650736s) [upload=91621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 09:23:08.760094669 (87h57m3.1911748s) [upload=91621]: Next segment err: metaclient: Too Many Requests Aug 12 09:23:08.760114537 (87h57m3.191194668s) [upload=91621]: Done uploading segments... Aug 12 09:23:08.760127261 (87h57m3.191207392s) [upload=91621]: Upload finished. err: metaclient: Too Many Requests Duration: 5.976177801s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.......X........X......] duration: 1.929361099s FAIL (2025-08-12 09:36:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.929386007s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.280054755s FAIL (2025-08-12 09:35:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.280076307s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.406521455s FAIL (2025-08-12 09:35:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.40653949s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [X....................................X.......X........X.....] duration: 1.979572202s FAIL (2025-08-12 09:35:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.979595606s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X........X......X......................X....X..............] duration: 5.869982389s FAIL (2025-08-12 09:35:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:23:08.516266754 (87h57m2.947346885s) [upload=91621]: Starting upload Aug 12 09:23:08.516319585 (87h57m2.947399716s) [upload=91621]: Done starting upload Aug 12 09:23:08.516447219 (87h57m2.947527371s) [upload=91621]: Uploading segments... Aug 12 09:23:08.516526641 (87h57m2.947606772s) [upload=91621]: Got next segment. Inline: false Aug 12 09:23:08.516564794 (87h57m2.947644925s) [upload=91621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:23:08.516570595 (87h57m2.947650736s) [upload=91621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 09:23:08.760094669 (87h57m3.1911748s) [upload=91621]: Next segment err: metaclient: Too Many Requests Aug 12 09:23:08.760114537 (87h57m3.191194668s) [upload=91621]: Done uploading segments... Aug 12 09:23:08.760127261 (87h57m3.191207392s) [upload=91621]: Upload finished. err: metaclient: Too Many Requests Duration: 5.870009652s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.361502139s FAIL (2025-08-12 09:35:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.361525474s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.231845138s FAIL (2025-08-12 09:34:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.231866147s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X........X......X......................X....X.............] duration: 5.654616085s FAIL (2025-08-12 09:34:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:23:08.516266754 (87h57m2.947346885s) [upload=91621]: Starting upload Aug 12 09:23:08.516319585 (87h57m2.947399716s) [upload=91621]: Done starting upload Aug 12 09:23:08.516447219 (87h57m2.947527371s) [upload=91621]: Uploading segments... Aug 12 09:23:08.516526641 (87h57m2.947606772s) [upload=91621]: Got next segment. Inline: false Aug 12 09:23:08.516564794 (87h57m2.947644925s) [upload=91621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:23:08.516570595 (87h57m2.947650736s) [upload=91621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 09:23:08.760094669 (87h57m3.1911748s) [upload=91621]: Next segment err: metaclient: Too Many Requests Aug 12 09:23:08.760114537 (87h57m3.191194668s) [upload=91621]: Done uploading segments... Aug 12 09:23:08.760127261 (87h57m3.191207392s) [upload=91621]: Upload finished. err: metaclient: Too Many Requests Duration: 5.654642766s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.X....................................X.......X........X....] duration: 1.87557728s FAIL (2025-08-12 09:34:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.875604532s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.350561161s FAIL (2025-08-12 09:34:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.350577003s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.497110797s FAIL (2025-08-12 09:34:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.497141434s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..X....................................X.......X........X...] duration: 1.895897096s FAIL (2025-08-12 09:34:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.895927805s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.31921982s FAIL (2025-08-12 09:33:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.319240229s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X........X......X......................X....X............] duration: 5.767295127s FAIL (2025-08-12 09:33:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:23:08.516266754 (87h57m2.947346885s) [upload=91621]: Starting upload Aug 12 09:23:08.516319585 (87h57m2.947399716s) [upload=91621]: Done starting upload Aug 12 09:23:08.516447219 (87h57m2.947527371s) [upload=91621]: Uploading segments... Aug 12 09:23:08.516526641 (87h57m2.947606772s) [upload=91621]: Got next segment. Inline: false Aug 12 09:23:08.516564794 (87h57m2.947644925s) [upload=91621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:23:08.516570595 (87h57m2.947650736s) [upload=91621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 09:23:08.760094669 (87h57m3.1911748s) [upload=91621]: Next segment err: metaclient: Too Many Requests Aug 12 09:23:08.760114537 (87h57m3.191194668s) [upload=91621]: Done uploading segments... Aug 12 09:23:08.760127261 (87h57m3.191207392s) [upload=91621]: Upload finished. err: metaclient: Too Many Requests Duration: 5.767312822s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...X....................................X.......X........X..] duration: 1.989941917s FAIL (2025-08-12 09:33:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.989967205s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.343181314s FAIL (2025-08-12 09:33:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.343202446s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.572433842s FAIL (2025-08-12 09:33:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.572454771s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....X....................................X.......X........X.] duration: 1.835356783s FAIL (2025-08-12 09:33:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.835383143s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....X....................................X.......X........X] duration: 1.789624296s FAIL (2025-08-12 09:32:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil> Duration:* 1.789676396s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X........X......X......................X....X...........] duration: 5.780335963s FAIL (2025-08-12 09:32:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:23:08.516266754 (87h57m2.947346885s) [upload=91621]: Starting upload Aug 12 09:23:08.516319585 (87h57m2.947399716s) [upload=91621]: Done starting upload Aug 12 09:23:08.516447219 (87h57m2.947527371s) [upload=91621]: Uploading segments... Aug 12 09:23:08.516526641 (87h57m2.947606772s) [upload=91621]: Got next segment. Inline: false Aug 12 09:23:08.516564794 (87h57m2.947644925s) [upload=91621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:23:08.516570595 (87h57m2.947650736s) [upload=91621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 09:23:08.760094669 (87h57m3.1911748s) [upload=91621]: Next segment err: metaclient: Too Many Requests Aug 12 09:23:08.760114537 (87h57m3.191194668s) [upload=91621]: Done uploading segments... Aug 12 09:23:08.760127261 (87h57m3.191207392s) [upload=91621]: Upload finished. err: metaclient: Too Many Requests Duration: 5.78036017s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......X....................................X.......X........] duration: 1.859524779s FAIL (2025-08-12 09:32:16 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:26:03.373412323 (87h59m57.804492454s) [upload=91674]: Starting upload Aug 12 09:26:03.373459974 (87h59m57.804540105s) [upload=91674]: Done starting upload Aug 12 09:26:03.373513897 (87h59m57.804594028s) [upload=91674]: Uploading segments... Aug 12 09:26:03.373573772 (87h59m57.804653903s) [upload=91674]: Got next segment. Inline: true Aug 12 09:26:03.373579032 (87h59m57.804659163s) [upload=91674]: Waiting for error group managing segments... Aug 12 09:26:03.373615531 (87h59m57.804695662s) [upload=91674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:26:03.373620261 (87h59m57.804700392s) [upload=91674]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:26:03.373622966 (87h59m57.804703087s) [upload=91674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:26:03.791789064 (87h59m58.222869195s) [upload=91674]: Done uploading segments... Aug 12 09:26:03.791806668 (87h59m58.222886799s) [upload=91674]: Upload finished. err: <nil> Duration:* 1.859550068s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.323893274s FAIL (2025-08-12 09:32:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.323930135s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.457771775s FAIL (2025-08-12 09:31:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.457792305s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......X....................................X.......X.......] duration: 1.824544285s FAIL (2025-08-12 09:31:45 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:26:03.373412323 (87h59m57.804492454s) [upload=91674]: Starting upload Aug 12 09:26:03.373459974 (87h59m57.804540105s) [upload=91674]: Done starting upload Aug 12 09:26:03.373513897 (87h59m57.804594028s) [upload=91674]: Uploading segments... Aug 12 09:26:03.373573772 (87h59m57.804653903s) [upload=91674]: Got next segment. Inline: true Aug 12 09:26:03.373579032 (87h59m57.804659163s) [upload=91674]: Waiting for error group managing segments... Aug 12 09:26:03.373615531 (87h59m57.804695662s) [upload=91674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 09:26:03.373620261 (87h59m57.804700392s) [upload=91674]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 09:26:03.373622966 (87h59m57.804703087s) [upload=91674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 09:26:03.791789064 (87h59m58.222869195s) [upload=91674]: Done uploading segments... Aug 12 09:26:03.791806668 (87h59m58.222886799s) [upload=91674]: Upload finished. err: <nil> Duration:* 1.824591724s core uplink us-select-1 us1-placement-20 metainfo-projects

1
...
12782
12783
12784
12785
12786
...
14958