Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................XX.X...............................] duration: 6.10022074s FAIL (2025-08-07 23:43:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 6.100246672s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......................XXXXXX.X.X............................] duration: 5.610229071s FAIL (2025-08-07 23:43:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.610252045s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................X.....X.X.......................] duration: 2.408526533s FAIL (2025-08-07 23:43:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.408548586s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [................X......X................................X...] duration: 1.871662645s FAIL (2025-08-07 23:43:00 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.871689036s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.655647476s FAIL (2025-08-07 23:42:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.655681833s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....................X.....................X..............X.] duration: 6.369939783s FAIL (2025-08-07 23:42:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 429484973 (14h1m26.01882996s) [noise=true , address=104.36.82.46:4486 , piece_id=C6ZAZJTHTBRFIPEJBA2M5X5W4AQNRYEHBEAZEQKOLRSZIVTGTBTQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.429745932 (14h1m26.019090909s) [noise=true , address=104.36.82.62:4430 , piece_id=7BJ4VANQHAR5WFIOOA2OJNS443L7DQL72POWSBZH6CH6XARICAOQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.430011039 (14h1m26.019356057s) [noise=true , address=104.36.82.16:4328 , piece_id=A6MVUJS5N43YTLGEHYXKOBHQZ2HPURDFTUD3C54BZJHN2XZKP6KA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.430755243 (14h1m26.02010024s) [noise=true , address=104.36.82.22:4529 , piece_id=37VR6ETFXFVQKINOKADIGEPC6GSV64BOHXCTP5SO3HACVFFK6XLQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.431116594 (14h1m26.020461572s) [noise=true , address=104.36.82.38:4705 , piece_id=62JG4LBNZUEAQTAF3H5R4WH64E5GBTFNXFL4CRFK3FMUY5S7A62A, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.431363697 (14h1m26.020708674s) [noise=true , address=104.36.82.17:4348 , piece_id=VD3CHZ2JYX4LNZPMLTORJKNZH5ZJ3HOFG3D43ZC3OKY5CRGFQBSQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.431608696 (14h1m26.020953683s) [noise=true , address=45.91.116.6:4995 , piece_id=2UDVI4FAORJL55O6DMARHJ5FUZDMAX437TPRK5DU4JLJO2W5HNZA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.431837944 (14h1m26.021182921s) [noise=true , address=79.127.205.245:4231 , piece_id=DXKQRFBREGYJVTS6N6DSIBC2B3E6WCEU56YCIDWP2HCSIZWOBM5Q, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.432139341 (14h1m26.021484328s) [noise=true , address=104.36.82.11:4147 , piece_id=CJFK5OBBXV4FOUIWAFXTDK34VSKJVMX4T3OCSHDPLV6GE5YKRVBQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.432355535 (14h1m26.021700522s) [noise=true , address=79.127.220.98:4418 , piece_id=I3XU6G4AIB4DK7LZAO5JWSBO5GYBDD6W4VSZXHHMYTEXVA3EOPGA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.432567942 (14h1m26.021912929s) [noise=true , address=104.36.82.36:4637 , piece_id=NEYMMM2HEN7LFGJ2PHZGX3NH5PUI3JPKJPFZRAKTEX3RFWCARMQA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.432771581 (14h1m26.022116558s) [noise=true , address=104.36.82.38:4691 , piece_id=HOYG2HVTZM45BK3SCODTKJQRZZ3BJ7KM23TXEEM7GFFBZEW5DF5Q, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.432976804 (14h1m26.022321791s) [noise=true , address=104.36.82.23:4451 , piece_id=SJN7C6MIO3AINEHZDXAXAFDSC6V6VOLMRKWS3YKWXSMIA7PVDIAA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.433183679 (14h1m26.022528666s) [noise=true , address=104.36.82.6:4878 , piece_id=X7232A4PYD2OMN2WUREWLFL44DG4K6CSBFIXX7YQEKJ77ETREIKA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.433383502 (14h1m26.022728489s) [noise=true , address=104.36.82.8:4994 , piece_id=MERUSSCDD6N2L3TD3ELJRVAZH564WLRIMCEZOWQHX2GIYIBSOBVQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.433629754 (14h1m26.022974731s) [noise=true , address=104.36.82.5:4866 , piece_id=KQ3L2NPCTMBXBOPW4O6TS6723PICYR36XVD626HHJULSKIQP757Q, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:27.541160437 (14h1m27.130505424s) [upload=14530]: Next returned nil segment Aug 7 23:41:27.541179232 (14h1m27.130524219s) [upload=14530]: Waiting for error group managing segments... Aug 7 23:41:27.834369491 (14h1m27.423714478s) [noise=true , address=79.127.205.245:4231 , piece_id=DXKQRFBREGYJVTS6N6DSIBC2B3E6WCEU56YCIDWP2HCSIZWOBM5Q, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.128318211 (14h1m27.717663199s) [noise=true , address=79.127.220.98:4418 , piece_id=I3XU6G4AIB4DK7LZAO5JWSBO5GYBDD6W4VSZXHHMYTEXVA3EOPGA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.387032228 (14h1m27.976377245s) [noise=true , address=104.36.82.22:4529 , piece_id=37VR6ETFXFVQKINOKADIGEPC6GSV64BOHXCTP5SO3HACVFFK6XLQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.405887072 (14h1m27.995232069s) [noise=true , address=104.36.82.8:4994 , piece_id=MERUSSCDD6N2L3TD3ELJRVAZH564WLRIMCEZOWQHX2GIYIBSOBVQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.409564789 (14h1m27.998909786s) [noise=true , address=45.91.116.6:4995 , piece_id=2UDVI4FAORJL55O6DMARHJ5FUZDMAX437TPRK5DU4JLJO2W5HNZA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.420829521 (14h1m28.010174518s) [noise=true , address=104.36.82.46:4486 , piece_id=C6ZAZJTHTBRFIPEJBA2M5X5W4AQNRYEHBEAZEQKOLRSZIVTGTBTQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.448432703 (14h1m28.03777769s) [noise=true , address=104.36.82.38:4691 , piece_id=HOYG2HVTZM45BK3SCODTKJQRZZ3BJ7KM23TXEEM7GFFBZEW5DF5Q, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.473580758 (14h1m28.062925745s) [noise=true , address=104.36.82.17:4348 , piece_id=VD3CHZ2JYX4LNZPMLTORJKNZH5ZJ3HOFG3D43ZC3OKY5CRGFQBSQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.517510246 (14h1m28.106855233s) [noise=true , address=104.36.82.11:4147 , piece_id=CJFK5OBBXV4FOUIWAFXTDK34VSKJVMX4T3OCSHDPLV6GE5YKRVBQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.531948059 (14h1m28.121293046s) [noise=true , address=104.36.82.38:4705 , piece_id=62JG4LBNZUEAQTAF3H5R4WH64E5GBTFNXFL4CRFK3FMUY5S7A62A, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.567804322 (14h1m28.157149309s) [noise=true , address=104.36.82.62:4430 , piece_id=7BJ4VANQHAR5WFIOOA2OJNS443L7DQL72POWSBZH6CH6XARICAOQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.606773701 (14h1m28.196118688s) [noise=true , address=104.36.82.6:4878 , piece_id=X7232A4PYD2OMN2WUREWLFL44DG4K6CSBFIXX7YQEKJ77ETREIKA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.687825593 (14h1m28.277170581s) [noise=true , address=104.36.82.36:4637 , piece_id=NEYMMM2HEN7LFGJ2PHZGX3NH5PUI3JPKJPFZRAKTEX3RFWCARMQA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:29.132992022 (14h1m28.722337019s) [noise=true , address=104.36.82.23:4451 , piece_id=SJN7C6MIO3AINEHZDXAXAFDSC6V6VOLMRKWS3YKWXSMIA7PVDIAA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:29.133039774 (14h1m28.722384771s) [seg_pos={0 0}, upload=14530]: Segment reached optimal threshold of 14 pieces. Aug 7 23:41:29.133659840 (14h1m28.723004817s) [noise=true , address=104.36.82.43:4864 , piece_id=KTJPZTWOZFPSEUKLVYR5RRGJ53DUWDMLJJSRUWO5L3W7UEDROPFA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1z6uPcGsR78PiEpT46NBJEdyH33Re1YU5CtrMMFVeFf1KD3egC): protocol: write tcp 188.166.225.170:58876->104.36.82.43:4864: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:41:29.133727991 (14h1m28.723072978s) [noise=true , address=104.36.82.59:4333 , piece_id=IR5BYHIL2N3MGSODPEA3QIARBJAAPW5ELH53Q2AQT5YSCTQOG3MQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K7U6NXcB6SFBtQFrG7DXt2skACr3u5DX2QsrnToi8EKywS9TN): protocol: read tcp 188.166.225.170:58514->104.36.82.59:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:41:29.133833823 (14h1m28.72317881s) [noise=true , address=104.36.82.16:4328 , piece_id=A6MVUJS5N43YTLGEHYXKOBHQZ2HPURDFTUD3C54BZJHN2XZKP6KA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ): protocol: rpc: dial tcp 104.36.82.16:4328: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:41:29.133843241 (14h1m28.723188228s) [noise=true , address=104.36.82.5:4866 , piece_id=KQ3L2NPCTMBXBOPW4O6TS6723PICYR36XVD626HHJULSKIQP757Q, seg_pos={0 0}, upload=14530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: write tcp 188.166.225.170:43022->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:41:29.133862939 (14h1m28.723207926s) [seg_pos={0 0}, upload=14530]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:41:29.133971787 (14h1m28.723316784s) [upload=14530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:41:29.133976927 (14h1m28.723321914s) [upload=14530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:29.440538157 (14h1m29.029883144s) [upload=14530]: Done uploading segments... Aug 7 23:41:29.440556182 (14h1m29.029901169s) [upload=14530]: Upload finished. err: <nil> Aug 7 23:42:13.801143611 (14h2m13.390488658s) [upload=14547]: Starting upload Aug 7 23:42:13.801245736 (14h2m13.390590723s) [upload=14547]: Done starting upload Aug 7 23:42:13.801335558 (14h2m13.390680545s) [upload=14547]: Uploading segments... Aug 7 23:42:13.801634330 (14h2m13.390979317s) [upload=14547]: Got next segment. Inline: false Aug 7 23:42:13.801664026 (14h2m13.391009013s) [upload=14547]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:42:13.801668035 (14h2m13.391013022s) [upload=14547]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:42:14.015121657 (14h2m13.604466644s) [upload=14547]: Next segment err: metaclient: Too Many Requests Aug 7 23:42:14.015150081 (14h2m13.604495068s) [upload=14547]: Done uploading segments... Aug 7 23:42:14.015187622 (14h2m13.604532609s) [upload=14547]: Upload finished. err: metaclient: Too Many Requests Duration: 6.369976513s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................XXXXX...................................] duration: 2.057381683s FAIL (2025-08-07 23:42:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.05740514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......................XXXX.................................] duration: 1.886541754s FAIL (2025-08-07 23:42:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.886572662s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [....................X...X..X.X..............................] duration: 2.042051644s FAIL (2025-08-07 23:42:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.04207627s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................XXXX.X..XX.............................] duration: 6.050037771s FAIL (2025-08-07 23:42:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.05005826s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 651.92949ms target: https://us1.storj.io/login FAIL (2025-08-07 23:42:30 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 651.955549ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................X......X................................X..] duration: 1.853553949s FAIL (2025-08-07 23:42:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.853586442s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................XXXX................................] duration: 1.892039671s FAIL (2025-08-07 23:42:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.892074348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....................X...X..X.X.............................] duration: 1.863659764s FAIL (2025-08-07 23:42:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.86368441s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.625166721s FAIL (2025-08-07 23:42:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.625189504s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................XXXXX..................................] duration: 2.018818031s FAIL (2025-08-07 23:42:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.018841747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....................X.....................X..............X] duration: 861.056923ms FAIL (2025-08-07 23:42:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 429484973 (14h1m26.01882996s) [noise=true , address=104.36.82.46:4486 , piece_id=C6ZAZJTHTBRFIPEJBA2M5X5W4AQNRYEHBEAZEQKOLRSZIVTGTBTQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.429745932 (14h1m26.019090909s) [noise=true , address=104.36.82.62:4430 , piece_id=7BJ4VANQHAR5WFIOOA2OJNS443L7DQL72POWSBZH6CH6XARICAOQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.430011039 (14h1m26.019356057s) [noise=true , address=104.36.82.16:4328 , piece_id=A6MVUJS5N43YTLGEHYXKOBHQZ2HPURDFTUD3C54BZJHN2XZKP6KA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.430755243 (14h1m26.02010024s) [noise=true , address=104.36.82.22:4529 , piece_id=37VR6ETFXFVQKINOKADIGEPC6GSV64BOHXCTP5SO3HACVFFK6XLQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.431116594 (14h1m26.020461572s) [noise=true , address=104.36.82.38:4705 , piece_id=62JG4LBNZUEAQTAF3H5R4WH64E5GBTFNXFL4CRFK3FMUY5S7A62A, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.431363697 (14h1m26.020708674s) [noise=true , address=104.36.82.17:4348 , piece_id=VD3CHZ2JYX4LNZPMLTORJKNZH5ZJ3HOFG3D43ZC3OKY5CRGFQBSQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.431608696 (14h1m26.020953683s) [noise=true , address=45.91.116.6:4995 , piece_id=2UDVI4FAORJL55O6DMARHJ5FUZDMAX437TPRK5DU4JLJO2W5HNZA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.431837944 (14h1m26.021182921s) [noise=true , address=79.127.205.245:4231 , piece_id=DXKQRFBREGYJVTS6N6DSIBC2B3E6WCEU56YCIDWP2HCSIZWOBM5Q, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.432139341 (14h1m26.021484328s) [noise=true , address=104.36.82.11:4147 , piece_id=CJFK5OBBXV4FOUIWAFXTDK34VSKJVMX4T3OCSHDPLV6GE5YKRVBQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.432355535 (14h1m26.021700522s) [noise=true , address=79.127.220.98:4418 , piece_id=I3XU6G4AIB4DK7LZAO5JWSBO5GYBDD6W4VSZXHHMYTEXVA3EOPGA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.432567942 (14h1m26.021912929s) [noise=true , address=104.36.82.36:4637 , piece_id=NEYMMM2HEN7LFGJ2PHZGX3NH5PUI3JPKJPFZRAKTEX3RFWCARMQA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.432771581 (14h1m26.022116558s) [noise=true , address=104.36.82.38:4691 , piece_id=HOYG2HVTZM45BK3SCODTKJQRZZ3BJ7KM23TXEEM7GFFBZEW5DF5Q, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.432976804 (14h1m26.022321791s) [noise=true , address=104.36.82.23:4451 , piece_id=SJN7C6MIO3AINEHZDXAXAFDSC6V6VOLMRKWS3YKWXSMIA7PVDIAA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.433183679 (14h1m26.022528666s) [noise=true , address=104.36.82.6:4878 , piece_id=X7232A4PYD2OMN2WUREWLFL44DG4K6CSBFIXX7YQEKJ77ETREIKA, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.433383502 (14h1m26.022728489s) [noise=true , address=104.36.82.8:4994 , piece_id=MERUSSCDD6N2L3TD3ELJRVAZH564WLRIMCEZOWQHX2GIYIBSOBVQ, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:26.433629754 (14h1m26.022974731s) [noise=true , address=104.36.82.5:4866 , piece_id=KQ3L2NPCTMBXBOPW4O6TS6723PICYR36XVD626HHJULSKIQP757Q, seg_pos={0 0}, upload=14530]: Uploading piece... Aug 7 23:41:27.541160437 (14h1m27.130505424s) [upload=14530]: Next returned nil segment Aug 7 23:41:27.541179232 (14h1m27.130524219s) [upload=14530]: Waiting for error group managing segments... Aug 7 23:41:27.834369491 (14h1m27.423714478s) [noise=true , address=79.127.205.245:4231 , piece_id=DXKQRFBREGYJVTS6N6DSIBC2B3E6WCEU56YCIDWP2HCSIZWOBM5Q, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.128318211 (14h1m27.717663199s) [noise=true , address=79.127.220.98:4418 , piece_id=I3XU6G4AIB4DK7LZAO5JWSBO5GYBDD6W4VSZXHHMYTEXVA3EOPGA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.387032228 (14h1m27.976377245s) [noise=true , address=104.36.82.22:4529 , piece_id=37VR6ETFXFVQKINOKADIGEPC6GSV64BOHXCTP5SO3HACVFFK6XLQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.405887072 (14h1m27.995232069s) [noise=true , address=104.36.82.8:4994 , piece_id=MERUSSCDD6N2L3TD3ELJRVAZH564WLRIMCEZOWQHX2GIYIBSOBVQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.409564789 (14h1m27.998909786s) [noise=true , address=45.91.116.6:4995 , piece_id=2UDVI4FAORJL55O6DMARHJ5FUZDMAX437TPRK5DU4JLJO2W5HNZA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.420829521 (14h1m28.010174518s) [noise=true , address=104.36.82.46:4486 , piece_id=C6ZAZJTHTBRFIPEJBA2M5X5W4AQNRYEHBEAZEQKOLRSZIVTGTBTQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.448432703 (14h1m28.03777769s) [noise=true , address=104.36.82.38:4691 , piece_id=HOYG2HVTZM45BK3SCODTKJQRZZ3BJ7KM23TXEEM7GFFBZEW5DF5Q, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.473580758 (14h1m28.062925745s) [noise=true , address=104.36.82.17:4348 , piece_id=VD3CHZ2JYX4LNZPMLTORJKNZH5ZJ3HOFG3D43ZC3OKY5CRGFQBSQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.517510246 (14h1m28.106855233s) [noise=true , address=104.36.82.11:4147 , piece_id=CJFK5OBBXV4FOUIWAFXTDK34VSKJVMX4T3OCSHDPLV6GE5YKRVBQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.531948059 (14h1m28.121293046s) [noise=true , address=104.36.82.38:4705 , piece_id=62JG4LBNZUEAQTAF3H5R4WH64E5GBTFNXFL4CRFK3FMUY5S7A62A, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.567804322 (14h1m28.157149309s) [noise=true , address=104.36.82.62:4430 , piece_id=7BJ4VANQHAR5WFIOOA2OJNS443L7DQL72POWSBZH6CH6XARICAOQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.606773701 (14h1m28.196118688s) [noise=true , address=104.36.82.6:4878 , piece_id=X7232A4PYD2OMN2WUREWLFL44DG4K6CSBFIXX7YQEKJ77ETREIKA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:28.687825593 (14h1m28.277170581s) [noise=true , address=104.36.82.36:4637 , piece_id=NEYMMM2HEN7LFGJ2PHZGX3NH5PUI3JPKJPFZRAKTEX3RFWCARMQA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:29.132992022 (14h1m28.722337019s) [noise=true , address=104.36.82.23:4451 , piece_id=SJN7C6MIO3AINEHZDXAXAFDSC6V6VOLMRKWS3YKWXSMIA7PVDIAA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: <nil> Aug 7 23:41:29.133039774 (14h1m28.722384771s) [seg_pos={0 0}, upload=14530]: Segment reached optimal threshold of 14 pieces. Aug 7 23:41:29.133659840 (14h1m28.723004817s) [noise=true , address=104.36.82.43:4864 , piece_id=KTJPZTWOZFPSEUKLVYR5RRGJ53DUWDMLJJSRUWO5L3W7UEDROPFA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1z6uPcGsR78PiEpT46NBJEdyH33Re1YU5CtrMMFVeFf1KD3egC): protocol: write tcp 188.166.225.170:58876->104.36.82.43:4864: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:41:29.133727991 (14h1m28.723072978s) [noise=true , address=104.36.82.59:4333 , piece_id=IR5BYHIL2N3MGSODPEA3QIARBJAAPW5ELH53Q2AQT5YSCTQOG3MQ, seg_pos={0 0}, upload=14530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K7U6NXcB6SFBtQFrG7DXt2skACr3u5DX2QsrnToi8EKywS9TN): protocol: read tcp 188.166.225.170:58514->104.36.82.59:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:41:29.133833823 (14h1m28.72317881s) [noise=true , address=104.36.82.16:4328 , piece_id=A6MVUJS5N43YTLGEHYXKOBHQZ2HPURDFTUD3C54BZJHN2XZKP6KA, seg_pos={0 0}, upload=14530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ): protocol: rpc: dial tcp 104.36.82.16:4328: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:41:29.133843241 (14h1m28.723188228s) [noise=true , address=104.36.82.5:4866 , piece_id=KQ3L2NPCTMBXBOPW4O6TS6723PICYR36XVD626HHJULSKIQP757Q, seg_pos={0 0}, upload=14530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: write tcp 188.166.225.170:43022->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:41:29.133862939 (14h1m28.723207926s) [seg_pos={0 0}, upload=14530]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:41:29.133971787 (14h1m28.723316784s) [upload=14530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:41:29.133976927 (14h1m28.723321914s) [upload=14530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:29.440538157 (14h1m29.029883144s) [upload=14530]: Done uploading segments... Aug 7 23:41:29.440556182 (14h1m29.029901169s) [upload=14530]: Upload finished. err: <nil> Aug 7 23:42:13.801143611 (14h2m13.390488658s) [upload=14547]: Starting upload Aug 7 23:42:13.801245736 (14h2m13.390590723s) [upload=14547]: Done starting upload Aug 7 23:42:13.801335558 (14h2m13.390680545s) [upload=14547]: Uploading segments... Aug 7 23:42:13.801634330 (14h2m13.390979317s) [upload=14547]: Got next segment. Inline: false Aug 7 23:42:13.801664026 (14h2m13.391009013s) [upload=14547]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:42:13.801668035 (14h2m13.391013022s) [upload=14547]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:42:14.015121657 (14h2m13.604466644s) [upload=14547]: Next segment err: metaclient: Too Many Requests Aug 7 23:42:14.015150081 (14h2m13.604495068s) [upload=14547]: Done uploading segments... Aug 7 23:42:14.015187622 (14h2m13.604532609s) [upload=14547]: Upload finished. err: metaclient: Too Many Requests Duration: 861.081421ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................X..X.XXXX.X...............................] duration: 6.485781489s FAIL (2025-08-07 23:42:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.485806507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XX.X..............................] duration: 6.70401657s FAIL (2025-08-07 23:42:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 6.704044934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................XXXXX.................................] duration: 2.032945753s FAIL (2025-08-07 23:41:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.032971091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.....X.X......................] duration: 2.344144524s FAIL (2025-08-07 23:41:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.344163922s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................X......X................................X.] duration: 1.842190717s FAIL (2025-08-07 23:41:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.842214072s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......................XXXXXX.X.X...........................] duration: 5.474995136s FAIL (2025-08-07 23:41:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.475016096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XX.X.............................] duration: 6.7667228s FAIL (2025-08-07 23:41:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 6.766744082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.716695258s FAIL (2025-08-07 23:41:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.716718634s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 652.114714ms target: https://us1.storj.io/login FAIL (2025-08-07 23:41:42 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 652.15456ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................XXXX.X..XX............................] duration: 5.925364956s FAIL (2025-08-07 23:41:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.925378992s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................X..X.XXXX.X..............................] duration: 7.454267376s FAIL (2025-08-07 23:41:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 7.454288234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X....................X.....................X..............] duration: 7.050366298s FAIL (2025-08-07 23:41:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 916s) [upload=14346]: Starting upload Aug 7 23:30:30.723725030 (13h50m30.313070017s) [upload=14346]: Done starting upload Aug 7 23:30:30.723796056 (13h50m30.313141043s) [upload=14346]: Uploading segments... Aug 7 23:30:30.723935803 (13h50m30.31328079s) [upload=14346]: Got next segment. Inline: false Aug 7 23:30:30.723960240 (13h50m30.313305227s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:30:30.723964218 (13h50m30.313309205s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:30:31.181251948 (13h50m30.770596926s) [seg_pos={0 0}, upload=14346]: Begin upload segment... Aug 7 23:30:31.181299649 (13h50m30.770644636s) [seg_pos={0 0}, upload=14346]: Done begin upload segment. Aug 7 23:30:31.181318165 (13h50m30.770663152s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181329696 (13h50m30.770674693s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181716817 (13h50m30.771061804s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181721437 (13h50m30.771066424s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182063421 (13h50m30.771408408s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182318108 (13h50m30.771663085s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182483654 (13h50m30.771828641s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182639132 (13h50m30.771984119s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182934928 (13h50m30.772279905s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183037023 (13h50m30.77238201s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183158877 (13h50m30.772503864s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183353329 (13h50m30.772698316s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183386993 (13h50m30.77273198s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183591444 (13h50m30.772936431s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183849538 (13h50m30.773194525s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184184719 (13h50m30.773529706s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184422474 (13h50m30.773767461s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184480636 (13h50m30.773825623s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:32.295332666 (13h50m31.884677653s) [upload=14346]: Next returned nil segment Aug 7 23:30:32.295347043 (13h50m31.88469204s) [upload=14346]: Waiting for error group managing segments... Aug 7 23:30:32.576560505 (13h50m32.165905492s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.584528981 (13h50m32.173873958s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.647385961 (13h50m32.236730948s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.094757008 (13h50m32.684101995s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.167659426 (13h50m32.757004403s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.198379320 (13h50m32.787724307s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.235504739 (13h50m32.824849726s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.264110248 (13h50m32.853455225s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.327068230 (13h50m32.916413217s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.366778193 (13h50m32.956123181s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.542767028 (13h50m33.132112016s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.575265094 (13h50m33.164610081s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.672325632 (13h50m33.261670619s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750650465 (13h50m33.339995452s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750672889 (13h50m33.340017876s) [seg_pos={0 0}, upload=14346]: Segment reached optimal threshold of 14 pieces. Aug 7 23:30:33.750785674 (13h50m33.340130661s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750885715 (13h50m33.340230702s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14FTsLbtv1vp43aRdDZZBY2Gf6epidM1A8HSyQKVokXGFFZGZk): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750979806 (13h50m33.340324793s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ui5Ye48cKNGFPakMKsjdBi13Xo7bzrmXPNEmRBLKwpMtVAA3R): protocol: rpc: dial tcp 104.36.82.62:4428: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:30:33.751073955 (13h50m33.340418942s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN): protocol: context canceled; context canceled Aug 7 23:30:33.751119673 (13h50m33.34046466s) [seg_pos={0 0}, upload=14346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:30:33.751224474 (13h50m33.340569471s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:30:33.751230776 (13h50m33.340575763s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:30:34.056540019 (13h50m33.645885006s) [upload=14346]: Done uploading segments... Aug 7 23:30:34.056559826 (13h50m33.645904823s) [upload=14346]: Upload finished. err: <nil> Duration: 7.050391656s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.867332679s FAIL (2025-08-07 23:41:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.867357125s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [........................XXXXXX.X.X..........................] duration: 5.600029849s FAIL (2025-08-07 23:41:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.600056441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................X...X..X.X............................] duration: 1.904271516s FAIL (2025-08-07 23:41:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.904298589s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................X......X................................X] duration: 1.620289434s FAIL (2025-08-07 23:41:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.62031405s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XX.X............................] duration: 7.972006852s FAIL (2025-08-07 23:41:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 7.972032051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........................XXXX...............................] duration: 1.867206029s FAIL (2025-08-07 23:41:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.867231468s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.....X.X.....................] duration: 2.596882719s FAIL (2025-08-07 23:40:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.596907446s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XX.X...........................] duration: 9.049403715s FAIL (2025-08-07 23:40:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 9.049455024s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................XXXX.X..XX...........................] duration: 6.26511048s FAIL (2025-08-07 23:40:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.265132151s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................XXXXX................................] duration: 1.900181019s FAIL (2025-08-07 23:40:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.900207749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......................X...X..X.X...........................] duration: 1.918969647s FAIL (2025-08-07 23:40:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.919018149s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 650.801562ms target: https://us1.storj.io/login FAIL (2025-08-07 23:40:47 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 650.818875ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................X..X.XXXX.X.............................] duration: 7.35722779s FAIL (2025-08-07 23:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 7.357307521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X....................X.....................X.............] duration: 6.793761776s FAIL (2025-08-07 23:40:39 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 916s) [upload=14346]: Starting upload Aug 7 23:30:30.723725030 (13h50m30.313070017s) [upload=14346]: Done starting upload Aug 7 23:30:30.723796056 (13h50m30.313141043s) [upload=14346]: Uploading segments... Aug 7 23:30:30.723935803 (13h50m30.31328079s) [upload=14346]: Got next segment. Inline: false Aug 7 23:30:30.723960240 (13h50m30.313305227s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:30:30.723964218 (13h50m30.313309205s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:30:31.181251948 (13h50m30.770596926s) [seg_pos={0 0}, upload=14346]: Begin upload segment... Aug 7 23:30:31.181299649 (13h50m30.770644636s) [seg_pos={0 0}, upload=14346]: Done begin upload segment. Aug 7 23:30:31.181318165 (13h50m30.770663152s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181329696 (13h50m30.770674693s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181716817 (13h50m30.771061804s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181721437 (13h50m30.771066424s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182063421 (13h50m30.771408408s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182318108 (13h50m30.771663085s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182483654 (13h50m30.771828641s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182639132 (13h50m30.771984119s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182934928 (13h50m30.772279905s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183037023 (13h50m30.77238201s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183158877 (13h50m30.772503864s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183353329 (13h50m30.772698316s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183386993 (13h50m30.77273198s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183591444 (13h50m30.772936431s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183849538 (13h50m30.773194525s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184184719 (13h50m30.773529706s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184422474 (13h50m30.773767461s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184480636 (13h50m30.773825623s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:32.295332666 (13h50m31.884677653s) [upload=14346]: Next returned nil segment Aug 7 23:30:32.295347043 (13h50m31.88469204s) [upload=14346]: Waiting for error group managing segments... Aug 7 23:30:32.576560505 (13h50m32.165905492s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.584528981 (13h50m32.173873958s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.647385961 (13h50m32.236730948s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.094757008 (13h50m32.684101995s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.167659426 (13h50m32.757004403s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.198379320 (13h50m32.787724307s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.235504739 (13h50m32.824849726s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.264110248 (13h50m32.853455225s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.327068230 (13h50m32.916413217s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.366778193 (13h50m32.956123181s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.542767028 (13h50m33.132112016s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.575265094 (13h50m33.164610081s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.672325632 (13h50m33.261670619s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750650465 (13h50m33.339995452s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750672889 (13h50m33.340017876s) [seg_pos={0 0}, upload=14346]: Segment reached optimal threshold of 14 pieces. Aug 7 23:30:33.750785674 (13h50m33.340130661s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750885715 (13h50m33.340230702s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14FTsLbtv1vp43aRdDZZBY2Gf6epidM1A8HSyQKVokXGFFZGZk): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750979806 (13h50m33.340324793s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ui5Ye48cKNGFPakMKsjdBi13Xo7bzrmXPNEmRBLKwpMtVAA3R): protocol: rpc: dial tcp 104.36.82.62:4428: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:30:33.751073955 (13h50m33.340418942s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN): protocol: context canceled; context canceled Aug 7 23:30:33.751119673 (13h50m33.34046466s) [seg_pos={0 0}, upload=14346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:30:33.751224474 (13h50m33.340569471s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:30:33.751230776 (13h50m33.340575763s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:30:34.056540019 (13h50m33.645885006s) [upload=14346]: Done uploading segments... Aug 7 23:30:34.056559826 (13h50m33.645904823s) [upload=14346]: Upload finished. err: <nil> Duration: 6.793787696s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.........................XXXXXX.X.X.........................] duration: 5.59118198s FAIL (2025-08-07 23:40:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.591238258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [..........................XXXX..............................] duration: 1.889069269s FAIL (2025-08-07 23:40:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.889099978s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................XXXX.X..XX..........................] duration: 6.400148922s FAIL (2025-08-07 23:40:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.400173388s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [........................XXXXX...............................] duration: 1.859848322s FAIL (2025-08-07 23:40:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.859874292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.973383587s FAIL (2025-08-07 23:40:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.973412892s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................X...X..X.X..........................] duration: 1.854416988s FAIL (2025-08-07 23:40:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.854447414s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 651.30294ms target: https://us1.storj.io/login FAIL (2025-08-07 23:40:16 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 651.329992ms core satellite us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)