Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 14.025499337s FAIL (2025-09-06 15:06:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 14.025520687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 19.248435918s FAIL (2025-09-06 15:05:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 19.248466638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX...X.X...XX] duration: 30.002341075s FAIL (2025-09-06 15:05:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:05:17.794909382 (1h36m53.834482502s) [upload=1402]: Starting upload Sep 6 15:05:17.794998452 (1h36m53.834571623s) [upload=1402]: Done starting upload Sep 6 15:05:17.795127179 (1h36m53.834700299s) [upload=1402]: Uploading segments... Sep 6 15:05:17.795213424 (1h36m53.834786544s) [upload=1402]: Got next segment. Inline: false Sep 6 15:05:17.795241247 (1h36m53.834814367s) [upload=1402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:05:17.795245786 (1h36m53.834818906s) [upload=1402]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:05:18.584956704 (1h36m54.624529825s) [seg_pos={0 0}, upload=1402]: Begin upload segment... Sep 6 15:05:18.584992473 (1h36m54.624565593s) [seg_pos={0 0}, upload=1402]: Done begin upload segment. Sep 6 15:05:18.585012411 (1h36m54.624585531s) [noise=true , address=45.91.116.11:4118 , piece_id=JD3TVPBMY54W2FVZRSYXB3FSU6QMUYOUATQLXYJKEY7IGZJP5CZQ, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.585092244 (1h36m54.624665364s) [noise=true , address=45.91.116.8:4048 , piece_id=FCALRGR5J7LUPEG3IXXHSOSBYBKR7P57W3PPVZCBZPKRACZFOU6A, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.585391837 (1h36m54.624964957s) [noise=true , address=45.91.116.13:4155 , piece_id=427CKI2NB2QBEI3TYVYBLGRJODPZA5BORIUAOXHPFWFOJV2HZMQQ, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.585503882 (1h36m54.625077002s) [noise=true , address=77.90.31.19:4843 , piece_id=7YZI42PHD4OQFGXU5XFX42XQTCELWG5M3KDEYUX56K35WPYGUX2A, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.585652666 (1h36m54.625225786s) [noise=true , address=45.91.116.5:4298 , piece_id=SZUWZCC6ISMSNMDKT5G26LVZDVOOK5OORGT2F5TSNXCOJ7RYI4NA, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.585779439 (1h36m54.625352559s) [noise=true , address=77.90.31.15:4698 , piece_id=Z4UYRVVKZJAXKUIVYDP6MO6GZ563DRAC5S65TQLSCA3OABRZ2FYQ, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.585963401 (1h36m54.625536521s) [noise=true , address=45.91.116.4:4937 , piece_id=3KBNCITZDHNHMEZUW3TXKIUXRR5237TW2ZX5FSXBH542MBVCSSZQ, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.586072830 (1h36m54.62564595s) [noise=true , address=77.90.31.17:4771 , piece_id=5XAJSVZZYBICVWFPMMGTDM6OJF7IFT7CBRTQEJPOUYH37VAXBIGQ, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.586235091 (1h36m54.625808201s) [noise=true , address=45.91.116.9:4261 , piece_id=ZHRAZBZL5GI7BT5WLAIXDNQLHO6HH7E75OZZ5JMFNEZUSZAHVN6Q, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.586376221 (1h36m54.625949341s) [noise=true , address=45.91.116.4:4938 , piece_id=4ECJ7E5Y5SU6QNELHIHSHSHCSEE4JWXPZUFX5V4S74QABCJUQBHQ, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.586492784 (1h36m54.626065894s) [noise=true , address=77.90.31.19:4841 , piece_id=WZZKTRTPPZS664JWFXGQU2UKOBMPLE33KX5DJ6L5LXI6MVMGXMIQ, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.586644915 (1h36m54.626218035s) [noise=true , address=45.91.116.4:4940 , piece_id=EDYI4Q6J7ZPHO7XS7C2RGXZTOSDYCNPQJ5OPFUPAJ32VZ33RRVTQ, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.586745427 (1h36m54.626318547s) [noise=true , address=45.91.116.7:4010 , piece_id=IOOIRVQQNXJOLBOPRWT6X2RGOZKH4GE77AN2PLP7GLG4YBE4WIFQ, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.586892729 (1h36m54.626465849s) [noise=true , address=45.91.116.6:4973 , piece_id=O67B2NWJQQJTLAZ24THYCBD5N7EJ4XJ5I45RISZZBEAYD6AGLTEQ, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.586993491 (1h36m54.626566611s) [noise=true , address=77.90.31.16:4735 , piece_id=3EAMHY6532FNFIJTKXF4IP5MEYVOY2SNSP6ZK672WLQ54AHW3A4Q, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:18.587244211 (1h36m54.626817321s) [noise=true , address=77.90.31.21:4878 , piece_id=7WHII5KUK2A3PEKCAPP7CRLWKWPGOKW2GCW53BY6GTGD5RKIA7PA, seg_pos={0 0}, upload=1402]: Uploading piece... Sep 6 15:05:20.437986836 (1h36m56.477559956s) [upload=1402]: Next returned nil segment Sep 6 15:05:20.438010752 (1h36m56.477583952s) [upload=1402]: Waiting for error group managing segments... Sep 6 15:05:20.730835419 (1h36m56.770408539s) [noise=true , address=45.91.116.11:4118 , piece_id=JD3TVPBMY54W2FVZRSYXB3FSU6QMUYOUATQLXYJKEY7IGZJP5CZQ, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:20.756566325 (1h36m56.796139435s) [noise=true , address=45.91.116.8:4048 , piece_id=FCALRGR5J7LUPEG3IXXHSOSBYBKR7P57W3PPVZCBZPKRACZFOU6A, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:20.764043413 (1h36m56.803616533s) [noise=true , address=45.91.116.4:4937 , piece_id=3KBNCITZDHNHMEZUW3TXKIUXRR5237TW2ZX5FSXBH542MBVCSSZQ, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:20.803422164 (1h36m56.842995284s) [noise=true , address=45.91.116.5:4298 , piece_id=SZUWZCC6ISMSNMDKT5G26LVZDVOOK5OORGT2F5TSNXCOJ7RYI4NA, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:20.824657740 (1h36m56.86423086s) [noise=true , address=45.91.116.13:4155 , piece_id=427CKI2NB2QBEI3TYVYBLGRJODPZA5BORIUAOXHPFWFOJV2HZMQQ, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:20.848998956 (1h36m56.888572076s) [noise=true , address=45.91.116.6:4973 , piece_id=O67B2NWJQQJTLAZ24THYCBD5N7EJ4XJ5I45RISZZBEAYD6AGLTEQ, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:20.850745839 (1h36m56.890318959s) [noise=true , address=45.91.116.4:4938 , piece_id=4ECJ7E5Y5SU6QNELHIHSHSHCSEE4JWXPZUFX5V4S74QABCJUQBHQ, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:20.854841627 (1h36m56.894414737s) [noise=true , address=77.90.31.21:4878 , piece_id=7WHII5KUK2A3PEKCAPP7CRLWKWPGOKW2GCW53BY6GTGD5RKIA7PA, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:20.870133097 (1h36m56.909706217s) [noise=true , address=77.90.31.15:4698 , piece_id=Z4UYRVVKZJAXKUIVYDP6MO6GZ563DRAC5S65TQLSCA3OABRZ2FYQ, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:20.872367022 (1h36m56.911940142s) [noise=true , address=45.91.116.4:4940 , piece_id=EDYI4Q6J7ZPHO7XS7C2RGXZTOSDYCNPQJ5OPFUPAJ32VZ33RRVTQ, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:20.881727123 (1h36m56.921300243s) [noise=true , address=77.90.31.16:4735 , piece_id=3EAMHY6532FNFIJTKXF4IP5MEYVOY2SNSP6ZK672WLQ54AHW3A4Q, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:20.962467842 (1h36m57.002040962s) [noise=true , address=45.91.116.9:4261 , piece_id=ZHRAZBZL5GI7BT5WLAIXDNQLHO6HH7E75OZZ5JMFNEZUSZAHVN6Q, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:21.098520130 (1h36m57.13809324s) [noise=true , address=45.91.116.7:4010 , piece_id=IOOIRVQQNXJOLBOPRWT6X2RGOZKH4GE77AN2PLP7GLG4YBE4WIFQ, seg_pos={0 0}, upload=1402]: Done uploading piece. err: <nil> Sep 6 15:05:21.098536641 (1h36m57.138109761s) [seg_pos={0 0}, upload=1402]: Segment reached optimal threshold of 13 pieces. Sep 6 15:05:21.098938130 (1h36m57.13851129s) [noise=true , address=77.90.31.17:4771 , piece_id=5XAJSVZZYBICVWFPMMGTDM6OJF7IFT7CBRTQEJPOUYH37VAXBIGQ, seg_pos={0 0}, upload=1402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:52314->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:52314->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52314->77.90.31.17:4771: use of closed network connection Sep 6 15:05:21.099007974 (1h36m57.138581094s) [noise=true , address=77.90.31.19:4843 , piece_id=7YZI42PHD4OQFGXU5XFX42XQTCELWG5M3KDEYUX56K35WPYGUX2A, seg_pos={0 0}, upload=1402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: write tcp 188.166.225.170:59104->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:59104->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59104->77.90.31.19:4843: use of closed network connection Sep 6 15:05:21.099049623 (1h36m57.138622743s) [noise=true , address=77.90.31.19:4841 , piece_id=WZZKTRTPPZS664JWFXGQU2UKOBMPLE33KX5DJ6L5LXI6MVMGXMIQ, seg_pos={0 0}, upload=1402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:59788->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:59788->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59788->77.90.31.19:4841: use of closed network connection Sep 6 15:05:21.099061977 (1h36m57.138635097s) [seg_pos={0 0}, upload=1402]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:05:21.099126110 (1h36m57.13869923s) [upload=1402]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:05:21.099129907 (1h36m57.138703017s) [upload=1402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:05:21.536905230 (1h36m57.57647835s) [upload=1402]: Done uploading segments... Sep 6 15:05:21.536924556 (1h36m57.576497676s) [upload=1402]: Upload finished. err: <nil> Duration: 30.002479601s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.239913456s FAIL (2025-09-06 15:05:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.239937963s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 13.227119158s FAIL (2025-09-06 15:04:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 13.227164496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX...X.X...X] duration: 30.001731134s FAIL (2025-09-06 15:04:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:04:03.500438162 (1h35m39.540011282s) [upload=1384]: Starting upload Sep 6 15:04:03.500524397 (1h35m39.540097517s) [upload=1384]: Done starting upload Sep 6 15:04:03.500613919 (1h35m39.540187049s) [upload=1384]: Uploading segments... Sep 6 15:04:03.500761782 (1h35m39.540334912s) [upload=1384]: Got next segment. Inline: false Sep 6 15:04:03.500805125 (1h35m39.540378245s) [upload=1384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:04:03.500812599 (1h35m39.540385739s) [upload=1384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:04:03.770671419 (1h35m39.810244539s) [seg_pos={0 0}, upload=1384]: Begin upload segment... Sep 6 15:04:03.770721716 (1h35m39.810294836s) [seg_pos={0 0}, upload=1384]: Done begin upload segment. Sep 6 15:04:03.770743658 (1h35m39.810316768s) [noise=true , address=77.90.31.13:4628 , piece_id=7LUYRF7Q54YGLM7AUB7LDPIBON7K6YBAD37B6G4H5ROZKPL2QZPQ, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.770782803 (1h35m39.810355974s) [noise=true , address=45.91.116.13:4153 , piece_id=T2TIPPDHRVRSFXVQR4N24RCHDSCLJHN52VVVY5ZL7KPYYRKYCJDQ, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.771129466 (1h35m39.810702586s) [noise=true , address=77.90.31.14:4663 , piece_id=F36F7SSR3BSJYRGJSV6BBKVFK24NR5KAXTSWWMOB46KYPHSQNAWA, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.771255468 (1h35m39.810828588s) [noise=true , address=45.91.116.7:4010 , piece_id=4ZULRG27H6AKB237GAGURMISAABNWEPHR342W6RRLVIKWX3UYAZA, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.771430342 (1h35m39.811003452s) [noise=true , address=77.90.31.15:4698 , piece_id=QEC6YZITFINX6FBI4AVVXFNTWTLOY4ZV5OO5FXF2NI33W7ZMTOKA, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.771703224 (1h35m39.811276344s) [noise=true , address=45.91.116.10:4081 , piece_id=MUYF52G2RNYGN7UWRCUXRNHR2WPA47LFHT6WA6UC7JTJHZAUG35Q, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.772059776 (1h35m39.811632886s) [noise=true , address=45.91.116.14:4190 , piece_id=KN6ULFFFFQ3DYU4OUDA6IBV44QOJBUPLZCGG2Q5CBR4OTC7FX5OA, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.772364750 (1h35m39.81193787s) [noise=true , address=77.90.31.19:4843 , piece_id=JIKOPZVRMRNQ7OW3QGY7YFWXLAK6VXSKGH7EVXDPAO5HJSZZDYSA, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.772586184 (1h35m39.812159304s) [noise=true , address=77.90.31.10:4518 , piece_id=R5ZMYRPRSZ4UOJ2HSPFPWMLCAF6KWA6IQREZNJENYNKB2CU7GTVA, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.772844749 (1h35m39.812417859s) [noise=true , address=45.91.116.12:4226 , piece_id=72WZIHWP45Z7CETXNYXPHMYW4GOWIWYB2UE6QIONYHGAFG2VWZQQ, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.773066703 (1h35m39.812639823s) [noise=true , address=45.91.116.4:4938 , piece_id=DOZA2XYIZPVSVUJUUUIGKSFO4E5AULELJ7D6AIZQXWD2SRIPWDWQ, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.773289048 (1h35m39.812862169s) [noise=true , address=77.90.31.10:4519 , piece_id=SVWQCQZAFJZKERGQ2P4MP4VZ47AWHZZCHR7XR7I5IWC6TR5BPWWA, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.773492037 (1h35m39.813065157s) [noise=true , address=77.90.31.17:4771 , piece_id=2WZIYB5NWLGD3YYS5KHBYUVRY5NLFJEKKCRP6COHF6MRZOGNDZ2Q, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.773684335 (1h35m39.813257455s) [noise=true , address=45.91.116.5:4298 , piece_id=NSARHIWPKJLGGYSJT4J5RP37BLQGOPDEGFAQUANWXGDIU4HUIN3A, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.773920567 (1h35m39.813493677s) [noise=true , address=45.91.116.13:4154 , piece_id=MCCEG67P7B2APDGERQ5ZIGUE6LGRGBXOSOXICKCNERRUSAR4UYDQ, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:03.774122243 (1h35m39.813695363s) [noise=true , address=45.91.116.12:4227 , piece_id=4UKA75MK7OWLYXYFQ772YKRBJJFCHRYFHOHDBD5WM3FHQ4RMEHEA, seg_pos={0 0}, upload=1384]: Uploading piece... Sep 6 15:04:05.587007325 (1h35m41.626580445s) [upload=1384]: Next returned nil segment Sep 6 15:04:05.587023627 (1h35m41.626596747s) [upload=1384]: Waiting for error group managing segments... Sep 6 15:04:05.961042429 (1h35m42.000615579s) [noise=true , address=45.91.116.14:4190 , piece_id=KN6ULFFFFQ3DYU4OUDA6IBV44QOJBUPLZCGG2Q5CBR4OTC7FX5OA, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:05.985877272 (1h35m42.025450402s) [noise=true , address=45.91.116.13:4153 , piece_id=T2TIPPDHRVRSFXVQR4N24RCHDSCLJHN52VVVY5ZL7KPYYRKYCJDQ, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:05.996622072 (1h35m42.036195192s) [noise=true , address=45.91.116.12:4227 , piece_id=4UKA75MK7OWLYXYFQ772YKRBJJFCHRYFHOHDBD5WM3FHQ4RMEHEA, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:05.997116608 (1h35m42.036689728s) [noise=true , address=45.91.116.4:4938 , piece_id=DOZA2XYIZPVSVUJUUUIGKSFO4E5AULELJ7D6AIZQXWD2SRIPWDWQ, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:06.004608695 (1h35m42.044181825s) [noise=true , address=45.91.116.13:4154 , piece_id=MCCEG67P7B2APDGERQ5ZIGUE6LGRGBXOSOXICKCNERRUSAR4UYDQ, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:06.007903590 (1h35m42.04747671s) [noise=true , address=45.91.116.5:4298 , piece_id=NSARHIWPKJLGGYSJT4J5RP37BLQGOPDEGFAQUANWXGDIU4HUIN3A, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:06.026596609 (1h35m42.066169729s) [noise=true , address=77.90.31.14:4663 , piece_id=F36F7SSR3BSJYRGJSV6BBKVFK24NR5KAXTSWWMOB46KYPHSQNAWA, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:06.042312832 (1h35m42.081885952s) [noise=true , address=45.91.116.12:4226 , piece_id=72WZIHWP45Z7CETXNYXPHMYW4GOWIWYB2UE6QIONYHGAFG2VWZQQ, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:06.096985158 (1h35m42.136558279s) [noise=true , address=45.91.116.7:4010 , piece_id=4ZULRG27H6AKB237GAGURMISAABNWEPHR342W6RRLVIKWX3UYAZA, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:06.102342740 (1h35m42.14191586s) [noise=true , address=77.90.31.19:4843 , piece_id=JIKOPZVRMRNQ7OW3QGY7YFWXLAK6VXSKGH7EVXDPAO5HJSZZDYSA, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:06.106785993 (1h35m42.146359113s) [noise=true , address=77.90.31.13:4628 , piece_id=7LUYRF7Q54YGLM7AUB7LDPIBON7K6YBAD37B6G4H5ROZKPL2QZPQ, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:06.107240964 (1h35m42.146814084s) [noise=true , address=77.90.31.15:4698 , piece_id=QEC6YZITFINX6FBI4AVVXFNTWTLOY4ZV5OO5FXF2NI33W7ZMTOKA, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:06.120950136 (1h35m42.160523256s) [noise=true , address=45.91.116.10:4081 , piece_id=MUYF52G2RNYGN7UWRCUXRNHR2WPA47LFHT6WA6UC7JTJHZAUG35Q, seg_pos={0 0}, upload=1384]: Done uploading piece. err: <nil> Sep 6 15:04:06.120973922 (1h35m42.160547042s) [seg_pos={0 0}, upload=1384]: Segment reached optimal threshold of 13 pieces. Sep 6 15:04:06.121080295 (1h35m42.160653415s) [noise=true , address=77.90.31.10:4519 , piece_id=SVWQCQZAFJZKERGQ2P4MP4VZ47AWHZZCHR7XR7I5IWC6TR5BPWWA, seg_pos={0 0}, upload=1384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Sep 6 15:04:06.121137675 (1h35m42.160710795s) [noise=true , address=77.90.31.17:4771 , piece_id=2WZIYB5NWLGD3YYS5KHBYUVRY5NLFJEKKCRP6COHF6MRZOGNDZ2Q, seg_pos={0 0}, upload=1384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 6 15:04:06.121226595 (1h35m42.160799715s) [noise=true , address=77.90.31.10:4518 , piece_id=R5ZMYRPRSZ4UOJ2HSPFPWMLCAF6KWA6IQREZNJENYNKB2CU7GTVA, seg_pos={0 0}, upload=1384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Sep 6 15:04:06.121252745 (1h35m42.160825865s) [seg_pos={0 0}, upload=1384]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:04:06.121328280 (1h35m42.1609014s) [upload=1384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:04:06.121332498 (1h35m42.160905618s) [upload=1384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:04:06.461539501 (1h35m42.501112621s) [upload=1384]: Done uploading segments... Sep 6 15:04:06.461566843 (1h35m42.501139963s) [upload=1384]: Upload finished. err: <nil> Duration: 30.001755961s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.519659553s FAIL (2025-09-06 15:04:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.519678099s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.707486925s FAIL (2025-09-06 15:03:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.707506192s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 14.854942189s FAIL (2025-09-06 15:03:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 14.854960794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [X..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX...X.X...] duration: 19.17506211s FAIL (2025-09-06 15:03:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:02:06.435203028 (1h33m42.474776149s) [upload=1353]: Starting upload Sep 6 15:02:06.435261470 (1h33m42.47483459s) [upload=1353]: Done starting upload Sep 6 15:02:06.435341413 (1h33m42.474914533s) [upload=1353]: Uploading segments... Sep 6 15:02:06.435423480 (1h33m42.4749966s) [upload=1353]: Got next segment. Inline: false Sep 6 15:02:06.435449731 (1h33m42.475022851s) [upload=1353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:02:06.435454790 (1h33m42.47502791s) [upload=1353]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:02:06.719134958 (1h33m42.758708078s) [seg_pos={0 0}, upload=1353]: Begin upload segment... Sep 6 15:02:06.719174073 (1h33m42.758747183s) [seg_pos={0 0}, upload=1353]: Done begin upload segment. Sep 6 15:02:06.719194071 (1h33m42.758767191s) [noise=true , address=77.90.31.12:4592 , piece_id=6L6ZSMQZSB7ZA4RKQ4J2XGOLLB2QKQ27Y3QCX3ELY3QS4OCYFJEQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719259317 (1h33m42.758832437s) [noise=true , address=45.91.116.6:4974 , piece_id=EHKWBKWXGHDCNOA2LFX7N623RJVCHXMO75SFUMD634QNAODQCDXQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719627781 (1h33m42.759200901s) [noise=true , address=45.91.116.9:4263 , piece_id=AXZAJ2HT6UAJ5B5LSQIV67OBSVSDUJFOU6R7UE23R2ZUEI2BSCKA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719689409 (1h33m42.759262529s) [noise=true , address=77.90.31.14:4662 , piece_id=PFPEY27X7KPFO6JHT5J7662LRCOHEWKCFNMLV57YXBA5DK7A6TGA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719936752 (1h33m42.759509872s) [noise=true , address=45.91.116.8:4047 , piece_id=DJMURSU3OBWUW6SH56UGVCPPXINJOEO65K2BN6BFHFASZHVJ6A3A, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720023329 (1h33m42.759596439s) [noise=true , address=77.90.31.19:4841 , piece_id=6SCRBKXTHHHV5HOAQAP6LO5LA3DARCP6AG2ED2J3ZQRXF37DZBOQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720284097 (1h33m42.759857217s) [noise=true , address=45.91.116.4:4940 , piece_id=MYUV4O5AEF7ILYIR256REDCETQSAT7IXI756WJ5YV3ZGTT6TWH2A, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720299307 (1h33m42.759872427s) [noise=true , address=45.91.116.9:4262 , piece_id=WBZDJBYUT3F3KGVSC7ESFW2VSSNKEHBNLU5EPZE7IUQIEHTB4PVQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720550928 (1h33m42.760124048s) [noise=true , address=77.90.31.21:4877 , piece_id=NNCC27QTLSFSD5AHHNRRO3Z3MHI644EQMZC2BFSTA5YMG77JIQXQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720556639 (1h33m42.760129759s) [noise=true , address=77.90.31.15:4699 , piece_id=K22QNDPA7RH7K4Y5YAU7WTYQXDEKKTFYS2RF2NMMPBVKHDDJUGFQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720798391 (1h33m42.760371511s) [noise=true , address=77.90.31.14:4663 , piece_id=HP26TEATXAMZ7ADVOY26Z5OLMSLY4EOSVZU4DCL5GZANGCRO2CYA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720816196 (1h33m42.760389316s) [noise=true , address=77.90.31.10:4519 , piece_id=LRAHMU2RHZWEJ7FWVADX7F2XC77755JAT2MF3JBM5DXH6TBHDERA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721064591 (1h33m42.760637711s) [noise=true , address=45.91.116.11:4119 , piece_id=BKTINOHEQ2O3HEJFRRFM7JSQK4FXRT2F4SNK76CAID7J7DX3ZFGQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721338214 (1h33m42.760911335s) [noise=true , address=45.91.116.5:4298 , piece_id=6HYNS73VU2SINOHQCUL6GAOENCQHOK767QVXIPVUMPTKAMTNYJUQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721634991 (1h33m42.761208113s) [noise=true , address=77.90.31.13:4628 , piece_id=N2HB7J6DTLDWORFGKEKTPW4BUB3FVXTXNDMD3QOHX5ONSQB4R2SQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.723224314 (1h33m42.762797434s) [noise=true , address=45.91.116.5:4297 , piece_id=ZSO6UTAZXMWAW7HPSZQNSINV5EJJCYWE3JUD2ZCY4MSYVSB4T6OQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:08.493939133 (1h33m44.533512253s) [upload=1353]: Next returned nil segment Sep 6 15:02:08.493956106 (1h33m44.533529226s) [upload=1353]: Waiting for error group managing segments... Sep 6 15:02:08.870919393 (1h33m44.910492513s) [noise=true , address=45.91.116.5:4297 , piece_id=ZSO6UTAZXMWAW7HPSZQNSINV5EJJCYWE3JUD2ZCY4MSYVSB4T6OQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.886315463 (1h33m44.925888583s) [noise=true , address=45.91.116.8:4047 , piece_id=DJMURSU3OBWUW6SH56UGVCPPXINJOEO65K2BN6BFHFASZHVJ6A3A, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.896559025 (1h33m44.936132145s) [noise=true , address=45.91.116.9:4262 , piece_id=WBZDJBYUT3F3KGVSC7ESFW2VSSNKEHBNLU5EPZE7IUQIEHTB4PVQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.919398439 (1h33m44.958971559s) [noise=true , address=45.91.116.11:4119 , piece_id=BKTINOHEQ2O3HEJFRRFM7JSQK4FXRT2F4SNK76CAID7J7DX3ZFGQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.927091850 (1h33m44.96666497s) [noise=true , address=45.91.116.5:4298 , piece_id=6HYNS73VU2SINOHQCUL6GAOENCQHOK767QVXIPVUMPTKAMTNYJUQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.939126108 (1h33m44.978699228s) [noise=true , address=45.91.116.9:4263 , piece_id=AXZAJ2HT6UAJ5B5LSQIV67OBSVSDUJFOU6R7UE23R2ZUEI2BSCKA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.961543394 (1h33m45.001116544s) [noise=true , address=77.90.31.10:4519 , piece_id=LRAHMU2RHZWEJ7FWVADX7F2XC77755JAT2MF3JBM5DXH6TBHDERA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.976047147 (1h33m45.015620267s) [noise=true , address=77.90.31.15:4699 , piece_id=K22QNDPA7RH7K4Y5YAU7WTYQXDEKKTFYS2RF2NMMPBVKHDDJUGFQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.991725177 (1h33m45.031298317s) [noise=true , address=77.90.31.13:4628 , piece_id=N2HB7J6DTLDWORFGKEKTPW4BUB3FVXTXNDMD3QOHX5ONSQB4R2SQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.996831889 (1h33m45.036405039s) [noise=true , address=45.91.116.6:4974 , piece_id=EHKWBKWXGHDCNOA2LFX7N623RJVCHXMO75SFUMD634QNAODQCDXQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.016987768 (1h33m45.056560888s) [noise=true , address=77.90.31.14:4663 , piece_id=HP26TEATXAMZ7ADVOY26Z5OLMSLY4EOSVZU4DCL5GZANGCRO2CYA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.020199874 (1h33m45.059772994s) [noise=true , address=77.90.31.12:4592 , piece_id=6L6ZSMQZSB7ZA4RKQ4J2XGOLLB2QKQ27Y3QCX3ELY3QS4OCYFJEQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.098200870 (1h33m45.13777399s) [noise=true , address=77.90.31.14:4662 , piece_id=PFPEY27X7KPFO6JHT5J7662LRCOHEWKCFNMLV57YXBA5DK7A6TGA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.098222312 (1h33m45.137795432s) [seg_pos={0 0}, upload=1353]: Segment reached optimal threshold of 13 pieces. Sep 6 15:02:09.098408117 (1h33m45.137981227s) [noise=true , address=45.91.116.4:4940 , piece_id=MYUV4O5AEF7ILYIR256REDCETQSAT7IXI756WJ5YV3ZGTT6TWH2A, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Sep 6 15:02:09.098466649 (1h33m45.138039769s) [noise=true , address=77.90.31.19:4841 , piece_id=6SCRBKXTHHHV5HOAQAP6LO5LA3DARCP6AG2ED2J3ZQRXF37DZBOQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Sep 6 15:02:09.098534228 (1h33m45.138107348s) [noise=true , address=77.90.31.21:4877 , piece_id=NNCC27QTLSFSD5AHHNRRO3Z3MHI644EQMZC2BFSTA5YMG77JIQXQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection Sep 6 15:02:09.098544048 (1h33m45.138117168s) [seg_pos={0 0}, upload=1353]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:02:09.098604794 (1h33m45.138177914s) [upload=1353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:02:09.098608410 (1h33m45.13818152s) [upload=1353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:02:10.429911787 (1h33m46.469484917s) [upload=1353]: Done uploading segments... Sep 6 15:02:10.429934030 (1h33m46.46950718s) [upload=1353]: Upload finished. err: <nil> Duration: 19.175087069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 32.07322421s FAIL (2025-09-06 15:03:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 32.073249318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX...X.X..] duration: 21.425076548s FAIL (2025-09-06 15:03:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:02:06.435203028 (1h33m42.474776149s) [upload=1353]: Starting upload Sep 6 15:02:06.435261470 (1h33m42.47483459s) [upload=1353]: Done starting upload Sep 6 15:02:06.435341413 (1h33m42.474914533s) [upload=1353]: Uploading segments... Sep 6 15:02:06.435423480 (1h33m42.4749966s) [upload=1353]: Got next segment. Inline: false Sep 6 15:02:06.435449731 (1h33m42.475022851s) [upload=1353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:02:06.435454790 (1h33m42.47502791s) [upload=1353]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:02:06.719134958 (1h33m42.758708078s) [seg_pos={0 0}, upload=1353]: Begin upload segment... Sep 6 15:02:06.719174073 (1h33m42.758747183s) [seg_pos={0 0}, upload=1353]: Done begin upload segment. Sep 6 15:02:06.719194071 (1h33m42.758767191s) [noise=true , address=77.90.31.12:4592 , piece_id=6L6ZSMQZSB7ZA4RKQ4J2XGOLLB2QKQ27Y3QCX3ELY3QS4OCYFJEQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719259317 (1h33m42.758832437s) [noise=true , address=45.91.116.6:4974 , piece_id=EHKWBKWXGHDCNOA2LFX7N623RJVCHXMO75SFUMD634QNAODQCDXQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719627781 (1h33m42.759200901s) [noise=true , address=45.91.116.9:4263 , piece_id=AXZAJ2HT6UAJ5B5LSQIV67OBSVSDUJFOU6R7UE23R2ZUEI2BSCKA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719689409 (1h33m42.759262529s) [noise=true , address=77.90.31.14:4662 , piece_id=PFPEY27X7KPFO6JHT5J7662LRCOHEWKCFNMLV57YXBA5DK7A6TGA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719936752 (1h33m42.759509872s) [noise=true , address=45.91.116.8:4047 , piece_id=DJMURSU3OBWUW6SH56UGVCPPXINJOEO65K2BN6BFHFASZHVJ6A3A, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720023329 (1h33m42.759596439s) [noise=true , address=77.90.31.19:4841 , piece_id=6SCRBKXTHHHV5HOAQAP6LO5LA3DARCP6AG2ED2J3ZQRXF37DZBOQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720284097 (1h33m42.759857217s) [noise=true , address=45.91.116.4:4940 , piece_id=MYUV4O5AEF7ILYIR256REDCETQSAT7IXI756WJ5YV3ZGTT6TWH2A, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720299307 (1h33m42.759872427s) [noise=true , address=45.91.116.9:4262 , piece_id=WBZDJBYUT3F3KGVSC7ESFW2VSSNKEHBNLU5EPZE7IUQIEHTB4PVQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720550928 (1h33m42.760124048s) [noise=true , address=77.90.31.21:4877 , piece_id=NNCC27QTLSFSD5AHHNRRO3Z3MHI644EQMZC2BFSTA5YMG77JIQXQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720556639 (1h33m42.760129759s) [noise=true , address=77.90.31.15:4699 , piece_id=K22QNDPA7RH7K4Y5YAU7WTYQXDEKKTFYS2RF2NMMPBVKHDDJUGFQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720798391 (1h33m42.760371511s) [noise=true , address=77.90.31.14:4663 , piece_id=HP26TEATXAMZ7ADVOY26Z5OLMSLY4EOSVZU4DCL5GZANGCRO2CYA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720816196 (1h33m42.760389316s) [noise=true , address=77.90.31.10:4519 , piece_id=LRAHMU2RHZWEJ7FWVADX7F2XC77755JAT2MF3JBM5DXH6TBHDERA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721064591 (1h33m42.760637711s) [noise=true , address=45.91.116.11:4119 , piece_id=BKTINOHEQ2O3HEJFRRFM7JSQK4FXRT2F4SNK76CAID7J7DX3ZFGQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721338214 (1h33m42.760911335s) [noise=true , address=45.91.116.5:4298 , piece_id=6HYNS73VU2SINOHQCUL6GAOENCQHOK767QVXIPVUMPTKAMTNYJUQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721634991 (1h33m42.761208113s) [noise=true , address=77.90.31.13:4628 , piece_id=N2HB7J6DTLDWORFGKEKTPW4BUB3FVXTXNDMD3QOHX5ONSQB4R2SQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.723224314 (1h33m42.762797434s) [noise=true , address=45.91.116.5:4297 , piece_id=ZSO6UTAZXMWAW7HPSZQNSINV5EJJCYWE3JUD2ZCY4MSYVSB4T6OQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:08.493939133 (1h33m44.533512253s) [upload=1353]: Next returned nil segment Sep 6 15:02:08.493956106 (1h33m44.533529226s) [upload=1353]: Waiting for error group managing segments... Sep 6 15:02:08.870919393 (1h33m44.910492513s) [noise=true , address=45.91.116.5:4297 , piece_id=ZSO6UTAZXMWAW7HPSZQNSINV5EJJCYWE3JUD2ZCY4MSYVSB4T6OQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.886315463 (1h33m44.925888583s) [noise=true , address=45.91.116.8:4047 , piece_id=DJMURSU3OBWUW6SH56UGVCPPXINJOEO65K2BN6BFHFASZHVJ6A3A, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.896559025 (1h33m44.936132145s) [noise=true , address=45.91.116.9:4262 , piece_id=WBZDJBYUT3F3KGVSC7ESFW2VSSNKEHBNLU5EPZE7IUQIEHTB4PVQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.919398439 (1h33m44.958971559s) [noise=true , address=45.91.116.11:4119 , piece_id=BKTINOHEQ2O3HEJFRRFM7JSQK4FXRT2F4SNK76CAID7J7DX3ZFGQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.927091850 (1h33m44.96666497s) [noise=true , address=45.91.116.5:4298 , piece_id=6HYNS73VU2SINOHQCUL6GAOENCQHOK767QVXIPVUMPTKAMTNYJUQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.939126108 (1h33m44.978699228s) [noise=true , address=45.91.116.9:4263 , piece_id=AXZAJ2HT6UAJ5B5LSQIV67OBSVSDUJFOU6R7UE23R2ZUEI2BSCKA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.961543394 (1h33m45.001116544s) [noise=true , address=77.90.31.10:4519 , piece_id=LRAHMU2RHZWEJ7FWVADX7F2XC77755JAT2MF3JBM5DXH6TBHDERA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.976047147 (1h33m45.015620267s) [noise=true , address=77.90.31.15:4699 , piece_id=K22QNDPA7RH7K4Y5YAU7WTYQXDEKKTFYS2RF2NMMPBVKHDDJUGFQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.991725177 (1h33m45.031298317s) [noise=true , address=77.90.31.13:4628 , piece_id=N2HB7J6DTLDWORFGKEKTPW4BUB3FVXTXNDMD3QOHX5ONSQB4R2SQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.996831889 (1h33m45.036405039s) [noise=true , address=45.91.116.6:4974 , piece_id=EHKWBKWXGHDCNOA2LFX7N623RJVCHXMO75SFUMD634QNAODQCDXQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.016987768 (1h33m45.056560888s) [noise=true , address=77.90.31.14:4663 , piece_id=HP26TEATXAMZ7ADVOY26Z5OLMSLY4EOSVZU4DCL5GZANGCRO2CYA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.020199874 (1h33m45.059772994s) [noise=true , address=77.90.31.12:4592 , piece_id=6L6ZSMQZSB7ZA4RKQ4J2XGOLLB2QKQ27Y3QCX3ELY3QS4OCYFJEQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.098200870 (1h33m45.13777399s) [noise=true , address=77.90.31.14:4662 , piece_id=PFPEY27X7KPFO6JHT5J7662LRCOHEWKCFNMLV57YXBA5DK7A6TGA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.098222312 (1h33m45.137795432s) [seg_pos={0 0}, upload=1353]: Segment reached optimal threshold of 13 pieces. Sep 6 15:02:09.098408117 (1h33m45.137981227s) [noise=true , address=45.91.116.4:4940 , piece_id=MYUV4O5AEF7ILYIR256REDCETQSAT7IXI756WJ5YV3ZGTT6TWH2A, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Sep 6 15:02:09.098466649 (1h33m45.138039769s) [noise=true , address=77.90.31.19:4841 , piece_id=6SCRBKXTHHHV5HOAQAP6LO5LA3DARCP6AG2ED2J3ZQRXF37DZBOQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Sep 6 15:02:09.098534228 (1h33m45.138107348s) [noise=true , address=77.90.31.21:4877 , piece_id=NNCC27QTLSFSD5AHHNRRO3Z3MHI644EQMZC2BFSTA5YMG77JIQXQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection Sep 6 15:02:09.098544048 (1h33m45.138117168s) [seg_pos={0 0}, upload=1353]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:02:09.098604794 (1h33m45.138177914s) [upload=1353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:02:09.098608410 (1h33m45.13818152s) [upload=1353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:02:10.429911787 (1h33m46.469484917s) [upload=1353]: Done uploading segments... Sep 6 15:02:10.429934030 (1h33m46.46950718s) [upload=1353]: Upload finished. err: <nil> Duration: 21.425090615s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.189614185s FAIL (2025-09-06 15:03:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.18964287s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX...X.X.] duration: 17.636474901s FAIL (2025-09-06 15:02:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:02:06.435203028 (1h33m42.474776149s) [upload=1353]: Starting upload Sep 6 15:02:06.435261470 (1h33m42.47483459s) [upload=1353]: Done starting upload Sep 6 15:02:06.435341413 (1h33m42.474914533s) [upload=1353]: Uploading segments... Sep 6 15:02:06.435423480 (1h33m42.4749966s) [upload=1353]: Got next segment. Inline: false Sep 6 15:02:06.435449731 (1h33m42.475022851s) [upload=1353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:02:06.435454790 (1h33m42.47502791s) [upload=1353]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:02:06.719134958 (1h33m42.758708078s) [seg_pos={0 0}, upload=1353]: Begin upload segment... Sep 6 15:02:06.719174073 (1h33m42.758747183s) [seg_pos={0 0}, upload=1353]: Done begin upload segment. Sep 6 15:02:06.719194071 (1h33m42.758767191s) [noise=true , address=77.90.31.12:4592 , piece_id=6L6ZSMQZSB7ZA4RKQ4J2XGOLLB2QKQ27Y3QCX3ELY3QS4OCYFJEQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719259317 (1h33m42.758832437s) [noise=true , address=45.91.116.6:4974 , piece_id=EHKWBKWXGHDCNOA2LFX7N623RJVCHXMO75SFUMD634QNAODQCDXQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719627781 (1h33m42.759200901s) [noise=true , address=45.91.116.9:4263 , piece_id=AXZAJ2HT6UAJ5B5LSQIV67OBSVSDUJFOU6R7UE23R2ZUEI2BSCKA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719689409 (1h33m42.759262529s) [noise=true , address=77.90.31.14:4662 , piece_id=PFPEY27X7KPFO6JHT5J7662LRCOHEWKCFNMLV57YXBA5DK7A6TGA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719936752 (1h33m42.759509872s) [noise=true , address=45.91.116.8:4047 , piece_id=DJMURSU3OBWUW6SH56UGVCPPXINJOEO65K2BN6BFHFASZHVJ6A3A, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720023329 (1h33m42.759596439s) [noise=true , address=77.90.31.19:4841 , piece_id=6SCRBKXTHHHV5HOAQAP6LO5LA3DARCP6AG2ED2J3ZQRXF37DZBOQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720284097 (1h33m42.759857217s) [noise=true , address=45.91.116.4:4940 , piece_id=MYUV4O5AEF7ILYIR256REDCETQSAT7IXI756WJ5YV3ZGTT6TWH2A, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720299307 (1h33m42.759872427s) [noise=true , address=45.91.116.9:4262 , piece_id=WBZDJBYUT3F3KGVSC7ESFW2VSSNKEHBNLU5EPZE7IUQIEHTB4PVQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720550928 (1h33m42.760124048s) [noise=true , address=77.90.31.21:4877 , piece_id=NNCC27QTLSFSD5AHHNRRO3Z3MHI644EQMZC2BFSTA5YMG77JIQXQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720556639 (1h33m42.760129759s) [noise=true , address=77.90.31.15:4699 , piece_id=K22QNDPA7RH7K4Y5YAU7WTYQXDEKKTFYS2RF2NMMPBVKHDDJUGFQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720798391 (1h33m42.760371511s) [noise=true , address=77.90.31.14:4663 , piece_id=HP26TEATXAMZ7ADVOY26Z5OLMSLY4EOSVZU4DCL5GZANGCRO2CYA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720816196 (1h33m42.760389316s) [noise=true , address=77.90.31.10:4519 , piece_id=LRAHMU2RHZWEJ7FWVADX7F2XC77755JAT2MF3JBM5DXH6TBHDERA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721064591 (1h33m42.760637711s) [noise=true , address=45.91.116.11:4119 , piece_id=BKTINOHEQ2O3HEJFRRFM7JSQK4FXRT2F4SNK76CAID7J7DX3ZFGQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721338214 (1h33m42.760911335s) [noise=true , address=45.91.116.5:4298 , piece_id=6HYNS73VU2SINOHQCUL6GAOENCQHOK767QVXIPVUMPTKAMTNYJUQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721634991 (1h33m42.761208113s) [noise=true , address=77.90.31.13:4628 , piece_id=N2HB7J6DTLDWORFGKEKTPW4BUB3FVXTXNDMD3QOHX5ONSQB4R2SQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.723224314 (1h33m42.762797434s) [noise=true , address=45.91.116.5:4297 , piece_id=ZSO6UTAZXMWAW7HPSZQNSINV5EJJCYWE3JUD2ZCY4MSYVSB4T6OQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:08.493939133 (1h33m44.533512253s) [upload=1353]: Next returned nil segment Sep 6 15:02:08.493956106 (1h33m44.533529226s) [upload=1353]: Waiting for error group managing segments... Sep 6 15:02:08.870919393 (1h33m44.910492513s) [noise=true , address=45.91.116.5:4297 , piece_id=ZSO6UTAZXMWAW7HPSZQNSINV5EJJCYWE3JUD2ZCY4MSYVSB4T6OQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.886315463 (1h33m44.925888583s) [noise=true , address=45.91.116.8:4047 , piece_id=DJMURSU3OBWUW6SH56UGVCPPXINJOEO65K2BN6BFHFASZHVJ6A3A, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.896559025 (1h33m44.936132145s) [noise=true , address=45.91.116.9:4262 , piece_id=WBZDJBYUT3F3KGVSC7ESFW2VSSNKEHBNLU5EPZE7IUQIEHTB4PVQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.919398439 (1h33m44.958971559s) [noise=true , address=45.91.116.11:4119 , piece_id=BKTINOHEQ2O3HEJFRRFM7JSQK4FXRT2F4SNK76CAID7J7DX3ZFGQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.927091850 (1h33m44.96666497s) [noise=true , address=45.91.116.5:4298 , piece_id=6HYNS73VU2SINOHQCUL6GAOENCQHOK767QVXIPVUMPTKAMTNYJUQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.939126108 (1h33m44.978699228s) [noise=true , address=45.91.116.9:4263 , piece_id=AXZAJ2HT6UAJ5B5LSQIV67OBSVSDUJFOU6R7UE23R2ZUEI2BSCKA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.961543394 (1h33m45.001116544s) [noise=true , address=77.90.31.10:4519 , piece_id=LRAHMU2RHZWEJ7FWVADX7F2XC77755JAT2MF3JBM5DXH6TBHDERA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.976047147 (1h33m45.015620267s) [noise=true , address=77.90.31.15:4699 , piece_id=K22QNDPA7RH7K4Y5YAU7WTYQXDEKKTFYS2RF2NMMPBVKHDDJUGFQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.991725177 (1h33m45.031298317s) [noise=true , address=77.90.31.13:4628 , piece_id=N2HB7J6DTLDWORFGKEKTPW4BUB3FVXTXNDMD3QOHX5ONSQB4R2SQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.996831889 (1h33m45.036405039s) [noise=true , address=45.91.116.6:4974 , piece_id=EHKWBKWXGHDCNOA2LFX7N623RJVCHXMO75SFUMD634QNAODQCDXQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.016987768 (1h33m45.056560888s) [noise=true , address=77.90.31.14:4663 , piece_id=HP26TEATXAMZ7ADVOY26Z5OLMSLY4EOSVZU4DCL5GZANGCRO2CYA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.020199874 (1h33m45.059772994s) [noise=true , address=77.90.31.12:4592 , piece_id=6L6ZSMQZSB7ZA4RKQ4J2XGOLLB2QKQ27Y3QCX3ELY3QS4OCYFJEQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.098200870 (1h33m45.13777399s) [noise=true , address=77.90.31.14:4662 , piece_id=PFPEY27X7KPFO6JHT5J7662LRCOHEWKCFNMLV57YXBA5DK7A6TGA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.098222312 (1h33m45.137795432s) [seg_pos={0 0}, upload=1353]: Segment reached optimal threshold of 13 pieces. Sep 6 15:02:09.098408117 (1h33m45.137981227s) [noise=true , address=45.91.116.4:4940 , piece_id=MYUV4O5AEF7ILYIR256REDCETQSAT7IXI756WJ5YV3ZGTT6TWH2A, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Sep 6 15:02:09.098466649 (1h33m45.138039769s) [noise=true , address=77.90.31.19:4841 , piece_id=6SCRBKXTHHHV5HOAQAP6LO5LA3DARCP6AG2ED2J3ZQRXF37DZBOQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Sep 6 15:02:09.098534228 (1h33m45.138107348s) [noise=true , address=77.90.31.21:4877 , piece_id=NNCC27QTLSFSD5AHHNRRO3Z3MHI644EQMZC2BFSTA5YMG77JIQXQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection Sep 6 15:02:09.098544048 (1h33m45.138117168s) [seg_pos={0 0}, upload=1353]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:02:09.098604794 (1h33m45.138177914s) [upload=1353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:02:09.098608410 (1h33m45.13818152s) [upload=1353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:02:10.429911787 (1h33m46.469484917s) [upload=1353]: Done uploading segments... Sep 6 15:02:10.429934030 (1h33m46.46950718s) [upload=1353]: Upload finished. err: <nil> Duration: 17.636490531s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.27964397s FAIL (2025-09-06 15:02:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.279654601s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX...X.X] duration: 30.002263425s FAIL (2025-09-06 15:02:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:02:06.435203028 (1h33m42.474776149s) [upload=1353]: Starting upload Sep 6 15:02:06.435261470 (1h33m42.47483459s) [upload=1353]: Done starting upload Sep 6 15:02:06.435341413 (1h33m42.474914533s) [upload=1353]: Uploading segments... Sep 6 15:02:06.435423480 (1h33m42.4749966s) [upload=1353]: Got next segment. Inline: false Sep 6 15:02:06.435449731 (1h33m42.475022851s) [upload=1353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:02:06.435454790 (1h33m42.47502791s) [upload=1353]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:02:06.719134958 (1h33m42.758708078s) [seg_pos={0 0}, upload=1353]: Begin upload segment... Sep 6 15:02:06.719174073 (1h33m42.758747183s) [seg_pos={0 0}, upload=1353]: Done begin upload segment. Sep 6 15:02:06.719194071 (1h33m42.758767191s) [noise=true , address=77.90.31.12:4592 , piece_id=6L6ZSMQZSB7ZA4RKQ4J2XGOLLB2QKQ27Y3QCX3ELY3QS4OCYFJEQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719259317 (1h33m42.758832437s) [noise=true , address=45.91.116.6:4974 , piece_id=EHKWBKWXGHDCNOA2LFX7N623RJVCHXMO75SFUMD634QNAODQCDXQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719627781 (1h33m42.759200901s) [noise=true , address=45.91.116.9:4263 , piece_id=AXZAJ2HT6UAJ5B5LSQIV67OBSVSDUJFOU6R7UE23R2ZUEI2BSCKA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719689409 (1h33m42.759262529s) [noise=true , address=77.90.31.14:4662 , piece_id=PFPEY27X7KPFO6JHT5J7662LRCOHEWKCFNMLV57YXBA5DK7A6TGA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.719936752 (1h33m42.759509872s) [noise=true , address=45.91.116.8:4047 , piece_id=DJMURSU3OBWUW6SH56UGVCPPXINJOEO65K2BN6BFHFASZHVJ6A3A, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720023329 (1h33m42.759596439s) [noise=true , address=77.90.31.19:4841 , piece_id=6SCRBKXTHHHV5HOAQAP6LO5LA3DARCP6AG2ED2J3ZQRXF37DZBOQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720284097 (1h33m42.759857217s) [noise=true , address=45.91.116.4:4940 , piece_id=MYUV4O5AEF7ILYIR256REDCETQSAT7IXI756WJ5YV3ZGTT6TWH2A, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720299307 (1h33m42.759872427s) [noise=true , address=45.91.116.9:4262 , piece_id=WBZDJBYUT3F3KGVSC7ESFW2VSSNKEHBNLU5EPZE7IUQIEHTB4PVQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720550928 (1h33m42.760124048s) [noise=true , address=77.90.31.21:4877 , piece_id=NNCC27QTLSFSD5AHHNRRO3Z3MHI644EQMZC2BFSTA5YMG77JIQXQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720556639 (1h33m42.760129759s) [noise=true , address=77.90.31.15:4699 , piece_id=K22QNDPA7RH7K4Y5YAU7WTYQXDEKKTFYS2RF2NMMPBVKHDDJUGFQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720798391 (1h33m42.760371511s) [noise=true , address=77.90.31.14:4663 , piece_id=HP26TEATXAMZ7ADVOY26Z5OLMSLY4EOSVZU4DCL5GZANGCRO2CYA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.720816196 (1h33m42.760389316s) [noise=true , address=77.90.31.10:4519 , piece_id=LRAHMU2RHZWEJ7FWVADX7F2XC77755JAT2MF3JBM5DXH6TBHDERA, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721064591 (1h33m42.760637711s) [noise=true , address=45.91.116.11:4119 , piece_id=BKTINOHEQ2O3HEJFRRFM7JSQK4FXRT2F4SNK76CAID7J7DX3ZFGQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721338214 (1h33m42.760911335s) [noise=true , address=45.91.116.5:4298 , piece_id=6HYNS73VU2SINOHQCUL6GAOENCQHOK767QVXIPVUMPTKAMTNYJUQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.721634991 (1h33m42.761208113s) [noise=true , address=77.90.31.13:4628 , piece_id=N2HB7J6DTLDWORFGKEKTPW4BUB3FVXTXNDMD3QOHX5ONSQB4R2SQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:06.723224314 (1h33m42.762797434s) [noise=true , address=45.91.116.5:4297 , piece_id=ZSO6UTAZXMWAW7HPSZQNSINV5EJJCYWE3JUD2ZCY4MSYVSB4T6OQ, seg_pos={0 0}, upload=1353]: Uploading piece... Sep 6 15:02:08.493939133 (1h33m44.533512253s) [upload=1353]: Next returned nil segment Sep 6 15:02:08.493956106 (1h33m44.533529226s) [upload=1353]: Waiting for error group managing segments... Sep 6 15:02:08.870919393 (1h33m44.910492513s) [noise=true , address=45.91.116.5:4297 , piece_id=ZSO6UTAZXMWAW7HPSZQNSINV5EJJCYWE3JUD2ZCY4MSYVSB4T6OQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.886315463 (1h33m44.925888583s) [noise=true , address=45.91.116.8:4047 , piece_id=DJMURSU3OBWUW6SH56UGVCPPXINJOEO65K2BN6BFHFASZHVJ6A3A, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.896559025 (1h33m44.936132145s) [noise=true , address=45.91.116.9:4262 , piece_id=WBZDJBYUT3F3KGVSC7ESFW2VSSNKEHBNLU5EPZE7IUQIEHTB4PVQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.919398439 (1h33m44.958971559s) [noise=true , address=45.91.116.11:4119 , piece_id=BKTINOHEQ2O3HEJFRRFM7JSQK4FXRT2F4SNK76CAID7J7DX3ZFGQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.927091850 (1h33m44.96666497s) [noise=true , address=45.91.116.5:4298 , piece_id=6HYNS73VU2SINOHQCUL6GAOENCQHOK767QVXIPVUMPTKAMTNYJUQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.939126108 (1h33m44.978699228s) [noise=true , address=45.91.116.9:4263 , piece_id=AXZAJ2HT6UAJ5B5LSQIV67OBSVSDUJFOU6R7UE23R2ZUEI2BSCKA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.961543394 (1h33m45.001116544s) [noise=true , address=77.90.31.10:4519 , piece_id=LRAHMU2RHZWEJ7FWVADX7F2XC77755JAT2MF3JBM5DXH6TBHDERA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.976047147 (1h33m45.015620267s) [noise=true , address=77.90.31.15:4699 , piece_id=K22QNDPA7RH7K4Y5YAU7WTYQXDEKKTFYS2RF2NMMPBVKHDDJUGFQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.991725177 (1h33m45.031298317s) [noise=true , address=77.90.31.13:4628 , piece_id=N2HB7J6DTLDWORFGKEKTPW4BUB3FVXTXNDMD3QOHX5ONSQB4R2SQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:08.996831889 (1h33m45.036405039s) [noise=true , address=45.91.116.6:4974 , piece_id=EHKWBKWXGHDCNOA2LFX7N623RJVCHXMO75SFUMD634QNAODQCDXQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.016987768 (1h33m45.056560888s) [noise=true , address=77.90.31.14:4663 , piece_id=HP26TEATXAMZ7ADVOY26Z5OLMSLY4EOSVZU4DCL5GZANGCRO2CYA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.020199874 (1h33m45.059772994s) [noise=true , address=77.90.31.12:4592 , piece_id=6L6ZSMQZSB7ZA4RKQ4J2XGOLLB2QKQ27Y3QCX3ELY3QS4OCYFJEQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.098200870 (1h33m45.13777399s) [noise=true , address=77.90.31.14:4662 , piece_id=PFPEY27X7KPFO6JHT5J7662LRCOHEWKCFNMLV57YXBA5DK7A6TGA, seg_pos={0 0}, upload=1353]: Done uploading piece. err: <nil> Sep 6 15:02:09.098222312 (1h33m45.137795432s) [seg_pos={0 0}, upload=1353]: Segment reached optimal threshold of 13 pieces. Sep 6 15:02:09.098408117 (1h33m45.137981227s) [noise=true , address=45.91.116.4:4940 , piece_id=MYUV4O5AEF7ILYIR256REDCETQSAT7IXI756WJ5YV3ZGTT6TWH2A, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Sep 6 15:02:09.098466649 (1h33m45.138039769s) [noise=true , address=77.90.31.19:4841 , piece_id=6SCRBKXTHHHV5HOAQAP6LO5LA3DARCP6AG2ED2J3ZQRXF37DZBOQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Sep 6 15:02:09.098534228 (1h33m45.138107348s) [noise=true , address=77.90.31.21:4877 , piece_id=NNCC27QTLSFSD5AHHNRRO3Z3MHI644EQMZC2BFSTA5YMG77JIQXQ, seg_pos={0 0}, upload=1353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.21:4877: use of closed network connection Sep 6 15:02:09.098544048 (1h33m45.138117168s) [seg_pos={0 0}, upload=1353]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:02:09.098604794 (1h33m45.138177914s) [upload=1353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:02:09.098608410 (1h33m45.13818152s) [upload=1353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:02:10.429911787 (1h33m46.469484917s) [upload=1353]: Done uploading segments... Sep 6 15:02:10.429934030 (1h33m46.46950718s) [upload=1353]: Upload finished. err: <nil> Duration: 30.002279647s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.843213301s FAIL (2025-09-06 15:02:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.843235804s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 10.175970098s FAIL (2025-09-06 15:02:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 10.175996178s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 19.943078928s FAIL (2025-09-06 15:01:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 19.943093357s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 7.080131481s FAIL (2025-09-06 15:01:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.080341362s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX...X.] duration: 24.631232535s FAIL (2025-09-06 15:01:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:00:29.162829075 (1h32m5.202402195s) [upload=1326]: Starting upload Sep 6 15:00:29.162885272 (1h32m5.202458392s) [upload=1326]: Done starting upload Sep 6 15:00:29.162937863 (1h32m5.202510983s) [upload=1326]: Uploading segments... Sep 6 15:00:29.163054597 (1h32m5.202627717s) [upload=1326]: Got next segment. Inline: false Sep 6 15:00:29.163090846 (1h32m5.202663966s) [upload=1326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:00:29.163095234 (1h32m5.202668354s) [upload=1326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:00:29.522799807 (1h32m5.562372927s) [seg_pos={0 0}, upload=1326]: Begin upload segment... Sep 6 15:00:29.522869329 (1h32m5.562442449s) [seg_pos={0 0}, upload=1326]: Done begin upload segment. Sep 6 15:00:29.522913895 (1h32m5.562487025s) [noise=true , address=45.91.116.9:4261 , piece_id=AFX5LENAY6VEYGLJGM6LRFKMMH4S4O3TPOILOWPKOMUG2ZTSFN3Q, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.522920547 (1h32m5.562493708s) [noise=true , address=45.91.116.11:4119 , piece_id=FYLZEDUQNWVACGO7YMEP36X2DNSGGSAR3AVNGY27FYV6MQO2E7QA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.523044335 (1h32m5.562617465s) [noise=true , address=77.90.31.10:4518 , piece_id=TOWXB3SLVCVZMKBDPYMNRFQ4SJ4LCVOB64SIMS63WUZDNGWY3RMQ, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.523667847 (1h32m5.563240977s) [noise=true , address=45.91.116.5:4297 , piece_id=6V3LADS7KDXS2J347VLIWUYDWT5GGI3A42VBP7SW54VQIL32CKEA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.523829667 (1h32m5.563402787s) [noise=true , address=45.91.116.5:4299 , piece_id=QEPT54DZ2HWHCIUEJLEUMPQ623SNB4VS2F5ZCML5RITSEHKYMD5Q, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.524253929 (1h32m5.563827079s) [noise=true , address=45.91.116.13:4155 , piece_id=PVJCNCMOWEMQZEE3DHNYF23NIR7QMXT6YX4QEE2ZZFDAAFWFXHXQ, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.524560976 (1h32m5.564134096s) [noise=true , address=45.91.116.4:4937 , piece_id=TK5EQIDDUE2Z4BKMGCSR7WU4DJP7R65FKJUY2DQKISG22ZLOV3WA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.524880809 (1h32m5.564453939s) [noise=true , address=45.91.116.4:4940 , piece_id=4GZ2MMFHXB4Y57LQ2MZAS7U32YXTG3YB2XMXBQX2FFMXDSHVUAWA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.525190721 (1h32m5.564763841s) [noise=true , address=77.90.31.21:4879 , piece_id=AVOPCHQ3S5FWLEPYIW635QHPKIOP5VSDOYK4JPQEXRDVBUQCBTLQ, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.525498240 (1h32m5.56507136s) [noise=true , address=45.91.116.14:4191 , piece_id=SKHOZYZA3AA73NZR6TNQ22VFCQEM2DDMNQEPO2FPJEQ4PWXY27HQ, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.525845424 (1h32m5.565418554s) [noise=true , address=45.91.116.8:4047 , piece_id=CVJSTNP34EQFCE4CMFGNHGCGXC2U3IFGN4XKREAIYCSCYLKUOS7Q, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.526264937 (1h32m5.565838057s) [noise=true , address=45.91.116.14:4189 , piece_id=YCDOJQUJSU763WEK7MALEGUM5F6XKEFOGLC5GELQOHLGEDC73YNA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.527596375 (1h32m5.567169505s) [noise=true , address=77.90.31.13:4628 , piece_id=2LKMXYJR64O3KQ7NSVXERD5BVX7IMQG2MNVMAZ3SULTYPT2TN4LQ, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.528110909 (1h32m5.567684029s) [noise=true , address=45.91.116.9:4263 , piece_id=45N3UW42M6A3URTUG4TR5SOJN6DF4GKR37UP7JCHVDZLGLYJRNVA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.528630323 (1h32m5.568203443s) [noise=true , address=77.90.31.17:4770 , piece_id=O2TQPGLCBEOT74OLYMKCABRBK5HHXGYKBNC7LUDP442L54KB4S3Q, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.529209912 (1h32m5.568783042s) [noise=true , address=45.91.116.13:4154 , piece_id=CWT274XMWZY2VPA7CDW6CS43OPRM73WJE6K4CSUOEGKREDMANG6A, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:31.303521484 (1h32m7.343094604s) [upload=1326]: Next returned nil segment Sep 6 15:00:31.303538006 (1h32m7.343111126s) [upload=1326]: Waiting for error group managing segments... Sep 6 15:00:31.700633997 (1h32m7.740207117s) [noise=true , address=77.90.31.21:4879 , piece_id=AVOPCHQ3S5FWLEPYIW635QHPKIOP5VSDOYK4JPQEXRDVBUQCBTLQ, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.736643783 (1h32m7.776216913s) [noise=true , address=77.90.31.10:4518 , piece_id=TOWXB3SLVCVZMKBDPYMNRFQ4SJ4LCVOB64SIMS63WUZDNGWY3RMQ, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.744688988 (1h32m7.784262108s) [noise=true , address=45.91.116.14:4189 , piece_id=YCDOJQUJSU763WEK7MALEGUM5F6XKEFOGLC5GELQOHLGEDC73YNA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.747949988 (1h32m7.787523098s) [noise=true , address=45.91.116.13:4155 , piece_id=PVJCNCMOWEMQZEE3DHNYF23NIR7QMXT6YX4QEE2ZZFDAAFWFXHXQ, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.755127964 (1h32m7.794701084s) [noise=true , address=45.91.116.9:4263 , piece_id=45N3UW42M6A3URTUG4TR5SOJN6DF4GKR37UP7JCHVDZLGLYJRNVA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.774401644 (1h32m7.813974774s) [noise=true , address=45.91.116.4:4937 , piece_id=TK5EQIDDUE2Z4BKMGCSR7WU4DJP7R65FKJUY2DQKISG22ZLOV3WA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.780004154 (1h32m7.819577274s) [noise=true , address=45.91.116.13:4154 , piece_id=CWT274XMWZY2VPA7CDW6CS43OPRM73WJE6K4CSUOEGKREDMANG6A, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.833490472 (1h32m7.873063592s) [noise=true , address=45.91.116.5:4297 , piece_id=6V3LADS7KDXS2J347VLIWUYDWT5GGI3A42VBP7SW54VQIL32CKEA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.862157136 (1h32m7.901730256s) [noise=true , address=45.91.116.11:4119 , piece_id=FYLZEDUQNWVACGO7YMEP36X2DNSGGSAR3AVNGY27FYV6MQO2E7QA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.866681353 (1h32m7.906254473s) [noise=true , address=77.90.31.13:4628 , piece_id=2LKMXYJR64O3KQ7NSVXERD5BVX7IMQG2MNVMAZ3SULTYPT2TN4LQ, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.887109112 (1h32m7.926682232s) [noise=true , address=45.91.116.14:4191 , piece_id=SKHOZYZA3AA73NZR6TNQ22VFCQEM2DDMNQEPO2FPJEQ4PWXY27HQ, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.920213768 (1h32m7.959786898s) [noise=true , address=45.91.116.8:4047 , piece_id=CVJSTNP34EQFCE4CMFGNHGCGXC2U3IFGN4XKREAIYCSCYLKUOS7Q, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.933420509 (1h32m7.972993629s) [noise=true , address=45.91.116.4:4940 , piece_id=4GZ2MMFHXB4Y57LQ2MZAS7U32YXTG3YB2XMXBQX2FFMXDSHVUAWA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.933446219 (1h32m7.973019339s) [seg_pos={0 0}, upload=1326]: Segment reached optimal threshold of 13 pieces. Sep 6 15:00:31.933596826 (1h32m7.973169946s) [noise=true , address=45.91.116.5:4299 , piece_id=QEPT54DZ2HWHCIUEJLEUMPQ623SNB4VS2F5ZCML5RITSEHKYMD5Q, seg_pos={0 0}, upload=1326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Sep 6 15:00:31.933667732 (1h32m7.973240852s) [noise=true , address=45.91.116.9:4261 , piece_id=AFX5LENAY6VEYGLJGM6LRFKMMH4S4O3TPOILOWPKOMUG2ZTSFN3Q, seg_pos={0 0}, upload=1326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Sep 6 15:00:31.933768916 (1h32m7.973342036s) [noise=true , address=77.90.31.17:4770 , piece_id=O2TQPGLCBEOT74OLYMKCABRBK5HHXGYKBNC7LUDP442L54KB4S3Q, seg_pos={0 0}, upload=1326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Sep 6 15:00:31.933840743 (1h32m7.973413863s) [seg_pos={0 0}, upload=1326]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:00:31.933947778 (1h32m7.973520898s) [upload=1326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:00:31.933953940 (1h32m7.97352706s) [upload=1326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:00:32.264074843 (1h32m8.303647963s) [upload=1326]: Done uploading segments... Sep 6 15:00:32.264098919 (1h32m8.303672049s) [upload=1326]: Upload finished. err: <nil> Duration: 24.631250109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 23.281439698s FAIL (2025-09-06 15:01:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 23.281459215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XXXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX...X] duration: 30.003760119s FAIL (2025-09-06 15:00:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:00:29.162829075 (1h32m5.202402195s) [upload=1326]: Starting upload Sep 6 15:00:29.162885272 (1h32m5.202458392s) [upload=1326]: Done starting upload Sep 6 15:00:29.162937863 (1h32m5.202510983s) [upload=1326]: Uploading segments... Sep 6 15:00:29.163054597 (1h32m5.202627717s) [upload=1326]: Got next segment. Inline: false Sep 6 15:00:29.163090846 (1h32m5.202663966s) [upload=1326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:00:29.163095234 (1h32m5.202668354s) [upload=1326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:00:29.522799807 (1h32m5.562372927s) [seg_pos={0 0}, upload=1326]: Begin upload segment... Sep 6 15:00:29.522869329 (1h32m5.562442449s) [seg_pos={0 0}, upload=1326]: Done begin upload segment. Sep 6 15:00:29.522913895 (1h32m5.562487025s) [noise=true , address=45.91.116.9:4261 , piece_id=AFX5LENAY6VEYGLJGM6LRFKMMH4S4O3TPOILOWPKOMUG2ZTSFN3Q, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.522920547 (1h32m5.562493708s) [noise=true , address=45.91.116.11:4119 , piece_id=FYLZEDUQNWVACGO7YMEP36X2DNSGGSAR3AVNGY27FYV6MQO2E7QA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.523044335 (1h32m5.562617465s) [noise=true , address=77.90.31.10:4518 , piece_id=TOWXB3SLVCVZMKBDPYMNRFQ4SJ4LCVOB64SIMS63WUZDNGWY3RMQ, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.523667847 (1h32m5.563240977s) [noise=true , address=45.91.116.5:4297 , piece_id=6V3LADS7KDXS2J347VLIWUYDWT5GGI3A42VBP7SW54VQIL32CKEA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.523829667 (1h32m5.563402787s) [noise=true , address=45.91.116.5:4299 , piece_id=QEPT54DZ2HWHCIUEJLEUMPQ623SNB4VS2F5ZCML5RITSEHKYMD5Q, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.524253929 (1h32m5.563827079s) [noise=true , address=45.91.116.13:4155 , piece_id=PVJCNCMOWEMQZEE3DHNYF23NIR7QMXT6YX4QEE2ZZFDAAFWFXHXQ, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.524560976 (1h32m5.564134096s) [noise=true , address=45.91.116.4:4937 , piece_id=TK5EQIDDUE2Z4BKMGCSR7WU4DJP7R65FKJUY2DQKISG22ZLOV3WA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.524880809 (1h32m5.564453939s) [noise=true , address=45.91.116.4:4940 , piece_id=4GZ2MMFHXB4Y57LQ2MZAS7U32YXTG3YB2XMXBQX2FFMXDSHVUAWA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.525190721 (1h32m5.564763841s) [noise=true , address=77.90.31.21:4879 , piece_id=AVOPCHQ3S5FWLEPYIW635QHPKIOP5VSDOYK4JPQEXRDVBUQCBTLQ, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.525498240 (1h32m5.56507136s) [noise=true , address=45.91.116.14:4191 , piece_id=SKHOZYZA3AA73NZR6TNQ22VFCQEM2DDMNQEPO2FPJEQ4PWXY27HQ, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.525845424 (1h32m5.565418554s) [noise=true , address=45.91.116.8:4047 , piece_id=CVJSTNP34EQFCE4CMFGNHGCGXC2U3IFGN4XKREAIYCSCYLKUOS7Q, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.526264937 (1h32m5.565838057s) [noise=true , address=45.91.116.14:4189 , piece_id=YCDOJQUJSU763WEK7MALEGUM5F6XKEFOGLC5GELQOHLGEDC73YNA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.527596375 (1h32m5.567169505s) [noise=true , address=77.90.31.13:4628 , piece_id=2LKMXYJR64O3KQ7NSVXERD5BVX7IMQG2MNVMAZ3SULTYPT2TN4LQ, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.528110909 (1h32m5.567684029s) [noise=true , address=45.91.116.9:4263 , piece_id=45N3UW42M6A3URTUG4TR5SOJN6DF4GKR37UP7JCHVDZLGLYJRNVA, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.528630323 (1h32m5.568203443s) [noise=true , address=77.90.31.17:4770 , piece_id=O2TQPGLCBEOT74OLYMKCABRBK5HHXGYKBNC7LUDP442L54KB4S3Q, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:29.529209912 (1h32m5.568783042s) [noise=true , address=45.91.116.13:4154 , piece_id=CWT274XMWZY2VPA7CDW6CS43OPRM73WJE6K4CSUOEGKREDMANG6A, seg_pos={0 0}, upload=1326]: Uploading piece... Sep 6 15:00:31.303521484 (1h32m7.343094604s) [upload=1326]: Next returned nil segment Sep 6 15:00:31.303538006 (1h32m7.343111126s) [upload=1326]: Waiting for error group managing segments... Sep 6 15:00:31.700633997 (1h32m7.740207117s) [noise=true , address=77.90.31.21:4879 , piece_id=AVOPCHQ3S5FWLEPYIW635QHPKIOP5VSDOYK4JPQEXRDVBUQCBTLQ, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.736643783 (1h32m7.776216913s) [noise=true , address=77.90.31.10:4518 , piece_id=TOWXB3SLVCVZMKBDPYMNRFQ4SJ4LCVOB64SIMS63WUZDNGWY3RMQ, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.744688988 (1h32m7.784262108s) [noise=true , address=45.91.116.14:4189 , piece_id=YCDOJQUJSU763WEK7MALEGUM5F6XKEFOGLC5GELQOHLGEDC73YNA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.747949988 (1h32m7.787523098s) [noise=true , address=45.91.116.13:4155 , piece_id=PVJCNCMOWEMQZEE3DHNYF23NIR7QMXT6YX4QEE2ZZFDAAFWFXHXQ, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.755127964 (1h32m7.794701084s) [noise=true , address=45.91.116.9:4263 , piece_id=45N3UW42M6A3URTUG4TR5SOJN6DF4GKR37UP7JCHVDZLGLYJRNVA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.774401644 (1h32m7.813974774s) [noise=true , address=45.91.116.4:4937 , piece_id=TK5EQIDDUE2Z4BKMGCSR7WU4DJP7R65FKJUY2DQKISG22ZLOV3WA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.780004154 (1h32m7.819577274s) [noise=true , address=45.91.116.13:4154 , piece_id=CWT274XMWZY2VPA7CDW6CS43OPRM73WJE6K4CSUOEGKREDMANG6A, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.833490472 (1h32m7.873063592s) [noise=true , address=45.91.116.5:4297 , piece_id=6V3LADS7KDXS2J347VLIWUYDWT5GGI3A42VBP7SW54VQIL32CKEA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.862157136 (1h32m7.901730256s) [noise=true , address=45.91.116.11:4119 , piece_id=FYLZEDUQNWVACGO7YMEP36X2DNSGGSAR3AVNGY27FYV6MQO2E7QA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.866681353 (1h32m7.906254473s) [noise=true , address=77.90.31.13:4628 , piece_id=2LKMXYJR64O3KQ7NSVXERD5BVX7IMQG2MNVMAZ3SULTYPT2TN4LQ, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.887109112 (1h32m7.926682232s) [noise=true , address=45.91.116.14:4191 , piece_id=SKHOZYZA3AA73NZR6TNQ22VFCQEM2DDMNQEPO2FPJEQ4PWXY27HQ, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.920213768 (1h32m7.959786898s) [noise=true , address=45.91.116.8:4047 , piece_id=CVJSTNP34EQFCE4CMFGNHGCGXC2U3IFGN4XKREAIYCSCYLKUOS7Q, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.933420509 (1h32m7.972993629s) [noise=true , address=45.91.116.4:4940 , piece_id=4GZ2MMFHXB4Y57LQ2MZAS7U32YXTG3YB2XMXBQX2FFMXDSHVUAWA, seg_pos={0 0}, upload=1326]: Done uploading piece. err: <nil> Sep 6 15:00:31.933446219 (1h32m7.973019339s) [seg_pos={0 0}, upload=1326]: Segment reached optimal threshold of 13 pieces. Sep 6 15:00:31.933596826 (1h32m7.973169946s) [noise=true , address=45.91.116.5:4299 , piece_id=QEPT54DZ2HWHCIUEJLEUMPQ623SNB4VS2F5ZCML5RITSEHKYMD5Q, seg_pos={0 0}, upload=1326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Sep 6 15:00:31.933667732 (1h32m7.973240852s) [noise=true , address=45.91.116.9:4261 , piece_id=AFX5LENAY6VEYGLJGM6LRFKMMH4S4O3TPOILOWPKOMUG2ZTSFN3Q, seg_pos={0 0}, upload=1326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Sep 6 15:00:31.933768916 (1h32m7.973342036s) [noise=true , address=77.90.31.17:4770 , piece_id=O2TQPGLCBEOT74OLYMKCABRBK5HHXGYKBNC7LUDP442L54KB4S3Q, seg_pos={0 0}, upload=1326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Sep 6 15:00:31.933840743 (1h32m7.973413863s) [seg_pos={0 0}, upload=1326]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:00:31.933947778 (1h32m7.973520898s) [upload=1326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:00:31.933953940 (1h32m7.97352706s) [upload=1326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:00:32.264074843 (1h32m8.303647963s) [upload=1326]: Done uploading segments... Sep 6 15:00:32.264098919 (1h32m8.303672049s) [upload=1326]: Upload finished. err: <nil> Duration: 30.003790498s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.722611978s FAIL (2025-09-06 15:00:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.72263365s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 42.788511672s FAIL (2025-09-06 15:00:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 42.788542269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.472500582s FAIL (2025-09-06 15:00:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.472543375s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [X.XXXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX...] duration: 28.21225255s FAIL (2025-09-06 15:00:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:57:24.998312993 (1h29m1.037886113s) [upload=1283]: Starting upload Sep 6 14:57:24.998368709 (1h29m1.037941829s) [upload=1283]: Done starting upload Sep 6 14:57:24.998455285 (1h29m1.038028405s) [upload=1283]: Uploading segments... Sep 6 14:57:24.998541089 (1h29m1.038114209s) [upload=1283]: Got next segment. Inline: false Sep 6 14:57:24.998564915 (1h29m1.038138055s) [upload=1283]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:57:24.998569193 (1h29m1.038142313s) [upload=1283]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:57:25.859008359 (1h29m1.898581479s) [seg_pos={0 0}, upload=1283]: Begin upload segment... Sep 6 14:57:25.859054547 (1h29m1.898627667s) [seg_pos={0 0}, upload=1283]: Done begin upload segment. Sep 6 14:57:25.859073945 (1h29m1.898647055s) [noise=true , address=77.90.31.15:4699 , piece_id=TEOVTYFY5LSVQ26AUQNBRUOF6HEADWJHUJWE7DKFUGWQLERLSF5A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859078483 (1h29m1.898651603s) [noise=true , address=45.91.116.4:4940 , piece_id=2QIMH2GXC5UTNUE2FZNMUZCE3UKWNBBUAKDQHZ64UXFTF2XDWA3A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859122246 (1h29m1.898695366s) [noise=true , address=77.90.31.19:4841 , piece_id=L5STZW6EZFZ4HBNEMPVUJVK6DF6H7UQDFGDCOVDA6MUXAZH4P6EQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859650578 (1h29m1.899223698s) [noise=true , address=77.90.31.14:4663 , piece_id=X4C654HA2AKA2M6OTHVOJDJ6OFEMAGSP6SA5PN6C26Z3KM6ER74Q, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860093936 (1h29m1.899667046s) [noise=true , address=45.91.116.7:4009 , piece_id=MHU26O5C5VLEQCD7MWUXPG7KXAMF6TSFEIA5ZS7JNQ6HJSXJJDWQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860372780 (1h29m1.8999459s) [noise=true , address=45.91.116.5:4298 , piece_id=RINQMF6FCCRXCA72TNWSMXYHFV3PQWWTQ5JRPXICONEEYZWRFHZA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860684226 (1h29m1.900257346s) [noise=true , address=45.91.116.6:4973 , piece_id=POJDDJEHVVNJSG46X654QZXXGNFTDB43CH5R4TEHLZ762V5TIENA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860935246 (1h29m1.900508366s) [noise=true , address=45.91.116.14:4191 , piece_id=IL6FLQPCQ6S5YJXU5VPNQEMYWZDMPJWQO6EJIQMMKN5SMO7D2JFQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861147522 (1h29m1.900720642s) [noise=true , address=45.91.116.7:4010 , piece_id=2XKNJ6DQA2Y73ZIR7FQSF5Y3JQPNMIU6DGFTJ6MJZNQYZA36EGWA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861349749 (1h29m1.900922869s) [noise=true , address=45.91.116.5:4297 , piece_id=QVXK6LGL2FI2YZGBEFBEC5VT2WFN4PJTZJVXZN5GMZBKRLNEL2XQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861551104 (1h29m1.901124224s) [noise=true , address=77.90.31.15:4698 , piece_id=GY2EOXKRNWWXZZMVUP4IAEMVEFKJWORCQU6K3JINKRLY5LASZK6A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861826782 (1h29m1.901399902s) [noise=true , address=45.91.116.9:4262 , piece_id=IAELUYAFEFMV4F2RPPPFPBXSJ57DTESG2NTYEJXHXJBWLVNFBMSA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862073464 (1h29m1.901646574s) [noise=true , address=77.90.31.12:4592 , piece_id=ES5NTXSG57ONAFVPZNIG45GR2JDNOSJHHK7M3UIWXTQSKEQSKXFQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862278476 (1h29m1.901851596s) [noise=true , address=45.91.116.13:4153 , piece_id=JEKLLAF4VLMNYDESDVGGRN3LMWH44JK6GLHP4HR4Y7K52W5VCYTA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862984698 (1h29m1.902557808s) [noise=true , address=45.91.116.4:4938 , piece_id=V7EUUJ57E6ULNT6NIFKDNW42OCLIPUYXXV6RJGKGO3HOIGDZ6ZOQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.863506877 (1h29m1.903079997s) [noise=true , address=77.90.31.18:4807 , piece_id=XFT4CDVMG5DSVPGT3AZK3IUXKIZAL2WAF63F6CA556QKNWZ7YPRQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:27.626500472 (1h29m3.666073592s) [upload=1283]: Next returned nil segment Sep 6 14:57:27.626522044 (1h29m3.666095164s) [upload=1283]: Waiting for error group managing segments... Sep 6 14:57:27.970829162 (1h29m4.010402282s) [noise=true , address=77.90.31.12:4592 , piece_id=ES5NTXSG57ONAFVPZNIG45GR2JDNOSJHHK7M3UIWXTQSKEQSKXFQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.014814389 (1h29m4.054387509s) [noise=true , address=45.91.116.9:4262 , piece_id=IAELUYAFEFMV4F2RPPPFPBXSJ57DTESG2NTYEJXHXJBWLVNFBMSA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.043386252 (1h29m4.082959362s) [noise=true , address=45.91.116.13:4153 , piece_id=JEKLLAF4VLMNYDESDVGGRN3LMWH44JK6GLHP4HR4Y7K52W5VCYTA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.052261515 (1h29m4.091834635s) [noise=true , address=77.90.31.19:4841 , piece_id=L5STZW6EZFZ4HBNEMPVUJVK6DF6H7UQDFGDCOVDA6MUXAZH4P6EQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.054781717 (1h29m4.094354838s) [noise=true , address=45.91.116.4:4940 , piece_id=2QIMH2GXC5UTNUE2FZNMUZCE3UKWNBBUAKDQHZ64UXFTF2XDWA3A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.088940631 (1h29m4.128513751s) [noise=true , address=77.90.31.14:4663 , piece_id=X4C654HA2AKA2M6OTHVOJDJ6OFEMAGSP6SA5PN6C26Z3KM6ER74Q, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.089269631 (1h29m4.128842761s) [noise=true , address=45.91.116.7:4010 , piece_id=2XKNJ6DQA2Y73ZIR7FQSF5Y3JQPNMIU6DGFTJ6MJZNQYZA36EGWA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.130989714 (1h29m4.170562834s) [noise=true , address=45.91.116.5:4298 , piece_id=RINQMF6FCCRXCA72TNWSMXYHFV3PQWWTQ5JRPXICONEEYZWRFHZA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.179333670 (1h29m4.21890679s) [noise=true , address=45.91.116.4:4938 , piece_id=V7EUUJ57E6ULNT6NIFKDNW42OCLIPUYXXV6RJGKGO3HOIGDZ6ZOQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.179384006 (1h29m4.218957116s) [noise=true , address=45.91.116.6:4973 , piece_id=POJDDJEHVVNJSG46X654QZXXGNFTDB43CH5R4TEHLZ762V5TIENA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.183564676 (1h29m4.223137796s) [noise=true , address=77.90.31.15:4699 , piece_id=TEOVTYFY5LSVQ26AUQNBRUOF6HEADWJHUJWE7DKFUGWQLERLSF5A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.220931549 (1h29m4.260504669s) [noise=true , address=45.91.116.7:4009 , piece_id=MHU26O5C5VLEQCD7MWUXPG7KXAMF6TSFEIA5ZS7JNQ6HJSXJJDWQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.373190019 (1h29m4.412763139s) [noise=true , address=77.90.31.15:4698 , piece_id=GY2EOXKRNWWXZZMVUP4IAEMVEFKJWORCQU6K3JINKRLY5LASZK6A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.373218523 (1h29m4.412791643s) [seg_pos={0 0}, upload=1283]: Segment reached optimal threshold of 13 pieces. Sep 6 14:57:28.373338613 (1h29m4.412911733s) [noise=true , address=45.91.116.5:4297 , piece_id=QVXK6LGL2FI2YZGBEFBEC5VT2WFN4PJTZJVXZN5GMZBKRLNEL2XQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Sep 6 14:57:28.373493670 (1h29m4.41306679s) [noise=true , address=45.91.116.14:4191 , piece_id=IL6FLQPCQ6S5YJXU5VPNQEMYWZDMPJWQO6EJIQMMKN5SMO7D2JFQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Sep 6 14:57:28.373590895 (1h29m4.413164015s) [noise=true , address=77.90.31.18:4807 , piece_id=XFT4CDVMG5DSVPGT3AZK3IUXKIZAL2WAF63F6CA556QKNWZ7YPRQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection Sep 6 14:57:28.373602157 (1h29m4.413175277s) [seg_pos={0 0}, upload=1283]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 14:57:28.373664106 (1h29m4.413237236s) [upload=1283]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:57:28.373710875 (1h29m4.413283996s) [upload=1283]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:57:28.850613317 (1h29m4.890186427s) [upload=1283]: Done uploading segments... Sep 6 14:57:28.850632084 (1h29m4.890205204s) [upload=1283]: Upload finished. err: <nil> Duration: 28.212287827s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.604901776s FAIL (2025-09-06 14:59:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.604927165s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 12.998977702s FAIL (2025-09-06 14:58:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 12.998993522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.X.XXXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX..] duration: 15.897004734s FAIL (2025-09-06 14:58:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:57:24.998312993 (1h29m1.037886113s) [upload=1283]: Starting upload Sep 6 14:57:24.998368709 (1h29m1.037941829s) [upload=1283]: Done starting upload Sep 6 14:57:24.998455285 (1h29m1.038028405s) [upload=1283]: Uploading segments... Sep 6 14:57:24.998541089 (1h29m1.038114209s) [upload=1283]: Got next segment. Inline: false Sep 6 14:57:24.998564915 (1h29m1.038138055s) [upload=1283]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:57:24.998569193 (1h29m1.038142313s) [upload=1283]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:57:25.859008359 (1h29m1.898581479s) [seg_pos={0 0}, upload=1283]: Begin upload segment... Sep 6 14:57:25.859054547 (1h29m1.898627667s) [seg_pos={0 0}, upload=1283]: Done begin upload segment. Sep 6 14:57:25.859073945 (1h29m1.898647055s) [noise=true , address=77.90.31.15:4699 , piece_id=TEOVTYFY5LSVQ26AUQNBRUOF6HEADWJHUJWE7DKFUGWQLERLSF5A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859078483 (1h29m1.898651603s) [noise=true , address=45.91.116.4:4940 , piece_id=2QIMH2GXC5UTNUE2FZNMUZCE3UKWNBBUAKDQHZ64UXFTF2XDWA3A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859122246 (1h29m1.898695366s) [noise=true , address=77.90.31.19:4841 , piece_id=L5STZW6EZFZ4HBNEMPVUJVK6DF6H7UQDFGDCOVDA6MUXAZH4P6EQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859650578 (1h29m1.899223698s) [noise=true , address=77.90.31.14:4663 , piece_id=X4C654HA2AKA2M6OTHVOJDJ6OFEMAGSP6SA5PN6C26Z3KM6ER74Q, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860093936 (1h29m1.899667046s) [noise=true , address=45.91.116.7:4009 , piece_id=MHU26O5C5VLEQCD7MWUXPG7KXAMF6TSFEIA5ZS7JNQ6HJSXJJDWQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860372780 (1h29m1.8999459s) [noise=true , address=45.91.116.5:4298 , piece_id=RINQMF6FCCRXCA72TNWSMXYHFV3PQWWTQ5JRPXICONEEYZWRFHZA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860684226 (1h29m1.900257346s) [noise=true , address=45.91.116.6:4973 , piece_id=POJDDJEHVVNJSG46X654QZXXGNFTDB43CH5R4TEHLZ762V5TIENA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860935246 (1h29m1.900508366s) [noise=true , address=45.91.116.14:4191 , piece_id=IL6FLQPCQ6S5YJXU5VPNQEMYWZDMPJWQO6EJIQMMKN5SMO7D2JFQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861147522 (1h29m1.900720642s) [noise=true , address=45.91.116.7:4010 , piece_id=2XKNJ6DQA2Y73ZIR7FQSF5Y3JQPNMIU6DGFTJ6MJZNQYZA36EGWA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861349749 (1h29m1.900922869s) [noise=true , address=45.91.116.5:4297 , piece_id=QVXK6LGL2FI2YZGBEFBEC5VT2WFN4PJTZJVXZN5GMZBKRLNEL2XQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861551104 (1h29m1.901124224s) [noise=true , address=77.90.31.15:4698 , piece_id=GY2EOXKRNWWXZZMVUP4IAEMVEFKJWORCQU6K3JINKRLY5LASZK6A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861826782 (1h29m1.901399902s) [noise=true , address=45.91.116.9:4262 , piece_id=IAELUYAFEFMV4F2RPPPFPBXSJ57DTESG2NTYEJXHXJBWLVNFBMSA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862073464 (1h29m1.901646574s) [noise=true , address=77.90.31.12:4592 , piece_id=ES5NTXSG57ONAFVPZNIG45GR2JDNOSJHHK7M3UIWXTQSKEQSKXFQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862278476 (1h29m1.901851596s) [noise=true , address=45.91.116.13:4153 , piece_id=JEKLLAF4VLMNYDESDVGGRN3LMWH44JK6GLHP4HR4Y7K52W5VCYTA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862984698 (1h29m1.902557808s) [noise=true , address=45.91.116.4:4938 , piece_id=V7EUUJ57E6ULNT6NIFKDNW42OCLIPUYXXV6RJGKGO3HOIGDZ6ZOQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.863506877 (1h29m1.903079997s) [noise=true , address=77.90.31.18:4807 , piece_id=XFT4CDVMG5DSVPGT3AZK3IUXKIZAL2WAF63F6CA556QKNWZ7YPRQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:27.626500472 (1h29m3.666073592s) [upload=1283]: Next returned nil segment Sep 6 14:57:27.626522044 (1h29m3.666095164s) [upload=1283]: Waiting for error group managing segments... Sep 6 14:57:27.970829162 (1h29m4.010402282s) [noise=true , address=77.90.31.12:4592 , piece_id=ES5NTXSG57ONAFVPZNIG45GR2JDNOSJHHK7M3UIWXTQSKEQSKXFQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.014814389 (1h29m4.054387509s) [noise=true , address=45.91.116.9:4262 , piece_id=IAELUYAFEFMV4F2RPPPFPBXSJ57DTESG2NTYEJXHXJBWLVNFBMSA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.043386252 (1h29m4.082959362s) [noise=true , address=45.91.116.13:4153 , piece_id=JEKLLAF4VLMNYDESDVGGRN3LMWH44JK6GLHP4HR4Y7K52W5VCYTA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.052261515 (1h29m4.091834635s) [noise=true , address=77.90.31.19:4841 , piece_id=L5STZW6EZFZ4HBNEMPVUJVK6DF6H7UQDFGDCOVDA6MUXAZH4P6EQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.054781717 (1h29m4.094354838s) [noise=true , address=45.91.116.4:4940 , piece_id=2QIMH2GXC5UTNUE2FZNMUZCE3UKWNBBUAKDQHZ64UXFTF2XDWA3A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.088940631 (1h29m4.128513751s) [noise=true , address=77.90.31.14:4663 , piece_id=X4C654HA2AKA2M6OTHVOJDJ6OFEMAGSP6SA5PN6C26Z3KM6ER74Q, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.089269631 (1h29m4.128842761s) [noise=true , address=45.91.116.7:4010 , piece_id=2XKNJ6DQA2Y73ZIR7FQSF5Y3JQPNMIU6DGFTJ6MJZNQYZA36EGWA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.130989714 (1h29m4.170562834s) [noise=true , address=45.91.116.5:4298 , piece_id=RINQMF6FCCRXCA72TNWSMXYHFV3PQWWTQ5JRPXICONEEYZWRFHZA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.179333670 (1h29m4.21890679s) [noise=true , address=45.91.116.4:4938 , piece_id=V7EUUJ57E6ULNT6NIFKDNW42OCLIPUYXXV6RJGKGO3HOIGDZ6ZOQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.179384006 (1h29m4.218957116s) [noise=true , address=45.91.116.6:4973 , piece_id=POJDDJEHVVNJSG46X654QZXXGNFTDB43CH5R4TEHLZ762V5TIENA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.183564676 (1h29m4.223137796s) [noise=true , address=77.90.31.15:4699 , piece_id=TEOVTYFY5LSVQ26AUQNBRUOF6HEADWJHUJWE7DKFUGWQLERLSF5A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.220931549 (1h29m4.260504669s) [noise=true , address=45.91.116.7:4009 , piece_id=MHU26O5C5VLEQCD7MWUXPG7KXAMF6TSFEIA5ZS7JNQ6HJSXJJDWQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.373190019 (1h29m4.412763139s) [noise=true , address=77.90.31.15:4698 , piece_id=GY2EOXKRNWWXZZMVUP4IAEMVEFKJWORCQU6K3JINKRLY5LASZK6A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.373218523 (1h29m4.412791643s) [seg_pos={0 0}, upload=1283]: Segment reached optimal threshold of 13 pieces. Sep 6 14:57:28.373338613 (1h29m4.412911733s) [noise=true , address=45.91.116.5:4297 , piece_id=QVXK6LGL2FI2YZGBEFBEC5VT2WFN4PJTZJVXZN5GMZBKRLNEL2XQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Sep 6 14:57:28.373493670 (1h29m4.41306679s) [noise=true , address=45.91.116.14:4191 , piece_id=IL6FLQPCQ6S5YJXU5VPNQEMYWZDMPJWQO6EJIQMMKN5SMO7D2JFQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Sep 6 14:57:28.373590895 (1h29m4.413164015s) [noise=true , address=77.90.31.18:4807 , piece_id=XFT4CDVMG5DSVPGT3AZK3IUXKIZAL2WAF63F6CA556QKNWZ7YPRQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection Sep 6 14:57:28.373602157 (1h29m4.413175277s) [seg_pos={0 0}, upload=1283]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 14:57:28.373664106 (1h29m4.413237236s) [upload=1283]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:57:28.373710875 (1h29m4.413283996s) [upload=1283]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:57:28.850613317 (1h29m4.890186427s) [upload=1283]: Done uploading segments... Sep 6 14:57:28.850632084 (1h29m4.890205204s) [upload=1283]: Upload finished. err: <nil> Duration: 15.897019673s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.970313549s FAIL (2025-09-06 14:58:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.970337664s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 25.269332789s FAIL (2025-09-06 14:58:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 25.269359229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [..X.XXXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX.] duration: 28.023963956s FAIL (2025-09-06 14:58:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:57:24.998312993 (1h29m1.037886113s) [upload=1283]: Starting upload Sep 6 14:57:24.998368709 (1h29m1.037941829s) [upload=1283]: Done starting upload Sep 6 14:57:24.998455285 (1h29m1.038028405s) [upload=1283]: Uploading segments... Sep 6 14:57:24.998541089 (1h29m1.038114209s) [upload=1283]: Got next segment. Inline: false Sep 6 14:57:24.998564915 (1h29m1.038138055s) [upload=1283]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:57:24.998569193 (1h29m1.038142313s) [upload=1283]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:57:25.859008359 (1h29m1.898581479s) [seg_pos={0 0}, upload=1283]: Begin upload segment... Sep 6 14:57:25.859054547 (1h29m1.898627667s) [seg_pos={0 0}, upload=1283]: Done begin upload segment. Sep 6 14:57:25.859073945 (1h29m1.898647055s) [noise=true , address=77.90.31.15:4699 , piece_id=TEOVTYFY5LSVQ26AUQNBRUOF6HEADWJHUJWE7DKFUGWQLERLSF5A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859078483 (1h29m1.898651603s) [noise=true , address=45.91.116.4:4940 , piece_id=2QIMH2GXC5UTNUE2FZNMUZCE3UKWNBBUAKDQHZ64UXFTF2XDWA3A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859122246 (1h29m1.898695366s) [noise=true , address=77.90.31.19:4841 , piece_id=L5STZW6EZFZ4HBNEMPVUJVK6DF6H7UQDFGDCOVDA6MUXAZH4P6EQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859650578 (1h29m1.899223698s) [noise=true , address=77.90.31.14:4663 , piece_id=X4C654HA2AKA2M6OTHVOJDJ6OFEMAGSP6SA5PN6C26Z3KM6ER74Q, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860093936 (1h29m1.899667046s) [noise=true , address=45.91.116.7:4009 , piece_id=MHU26O5C5VLEQCD7MWUXPG7KXAMF6TSFEIA5ZS7JNQ6HJSXJJDWQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860372780 (1h29m1.8999459s) [noise=true , address=45.91.116.5:4298 , piece_id=RINQMF6FCCRXCA72TNWSMXYHFV3PQWWTQ5JRPXICONEEYZWRFHZA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860684226 (1h29m1.900257346s) [noise=true , address=45.91.116.6:4973 , piece_id=POJDDJEHVVNJSG46X654QZXXGNFTDB43CH5R4TEHLZ762V5TIENA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860935246 (1h29m1.900508366s) [noise=true , address=45.91.116.14:4191 , piece_id=IL6FLQPCQ6S5YJXU5VPNQEMYWZDMPJWQO6EJIQMMKN5SMO7D2JFQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861147522 (1h29m1.900720642s) [noise=true , address=45.91.116.7:4010 , piece_id=2XKNJ6DQA2Y73ZIR7FQSF5Y3JQPNMIU6DGFTJ6MJZNQYZA36EGWA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861349749 (1h29m1.900922869s) [noise=true , address=45.91.116.5:4297 , piece_id=QVXK6LGL2FI2YZGBEFBEC5VT2WFN4PJTZJVXZN5GMZBKRLNEL2XQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861551104 (1h29m1.901124224s) [noise=true , address=77.90.31.15:4698 , piece_id=GY2EOXKRNWWXZZMVUP4IAEMVEFKJWORCQU6K3JINKRLY5LASZK6A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861826782 (1h29m1.901399902s) [noise=true , address=45.91.116.9:4262 , piece_id=IAELUYAFEFMV4F2RPPPFPBXSJ57DTESG2NTYEJXHXJBWLVNFBMSA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862073464 (1h29m1.901646574s) [noise=true , address=77.90.31.12:4592 , piece_id=ES5NTXSG57ONAFVPZNIG45GR2JDNOSJHHK7M3UIWXTQSKEQSKXFQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862278476 (1h29m1.901851596s) [noise=true , address=45.91.116.13:4153 , piece_id=JEKLLAF4VLMNYDESDVGGRN3LMWH44JK6GLHP4HR4Y7K52W5VCYTA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862984698 (1h29m1.902557808s) [noise=true , address=45.91.116.4:4938 , piece_id=V7EUUJ57E6ULNT6NIFKDNW42OCLIPUYXXV6RJGKGO3HOIGDZ6ZOQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.863506877 (1h29m1.903079997s) [noise=true , address=77.90.31.18:4807 , piece_id=XFT4CDVMG5DSVPGT3AZK3IUXKIZAL2WAF63F6CA556QKNWZ7YPRQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:27.626500472 (1h29m3.666073592s) [upload=1283]: Next returned nil segment Sep 6 14:57:27.626522044 (1h29m3.666095164s) [upload=1283]: Waiting for error group managing segments... Sep 6 14:57:27.970829162 (1h29m4.010402282s) [noise=true , address=77.90.31.12:4592 , piece_id=ES5NTXSG57ONAFVPZNIG45GR2JDNOSJHHK7M3UIWXTQSKEQSKXFQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.014814389 (1h29m4.054387509s) [noise=true , address=45.91.116.9:4262 , piece_id=IAELUYAFEFMV4F2RPPPFPBXSJ57DTESG2NTYEJXHXJBWLVNFBMSA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.043386252 (1h29m4.082959362s) [noise=true , address=45.91.116.13:4153 , piece_id=JEKLLAF4VLMNYDESDVGGRN3LMWH44JK6GLHP4HR4Y7K52W5VCYTA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.052261515 (1h29m4.091834635s) [noise=true , address=77.90.31.19:4841 , piece_id=L5STZW6EZFZ4HBNEMPVUJVK6DF6H7UQDFGDCOVDA6MUXAZH4P6EQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.054781717 (1h29m4.094354838s) [noise=true , address=45.91.116.4:4940 , piece_id=2QIMH2GXC5UTNUE2FZNMUZCE3UKWNBBUAKDQHZ64UXFTF2XDWA3A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.088940631 (1h29m4.128513751s) [noise=true , address=77.90.31.14:4663 , piece_id=X4C654HA2AKA2M6OTHVOJDJ6OFEMAGSP6SA5PN6C26Z3KM6ER74Q, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.089269631 (1h29m4.128842761s) [noise=true , address=45.91.116.7:4010 , piece_id=2XKNJ6DQA2Y73ZIR7FQSF5Y3JQPNMIU6DGFTJ6MJZNQYZA36EGWA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.130989714 (1h29m4.170562834s) [noise=true , address=45.91.116.5:4298 , piece_id=RINQMF6FCCRXCA72TNWSMXYHFV3PQWWTQ5JRPXICONEEYZWRFHZA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.179333670 (1h29m4.21890679s) [noise=true , address=45.91.116.4:4938 , piece_id=V7EUUJ57E6ULNT6NIFKDNW42OCLIPUYXXV6RJGKGO3HOIGDZ6ZOQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.179384006 (1h29m4.218957116s) [noise=true , address=45.91.116.6:4973 , piece_id=POJDDJEHVVNJSG46X654QZXXGNFTDB43CH5R4TEHLZ762V5TIENA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.183564676 (1h29m4.223137796s) [noise=true , address=77.90.31.15:4699 , piece_id=TEOVTYFY5LSVQ26AUQNBRUOF6HEADWJHUJWE7DKFUGWQLERLSF5A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.220931549 (1h29m4.260504669s) [noise=true , address=45.91.116.7:4009 , piece_id=MHU26O5C5VLEQCD7MWUXPG7KXAMF6TSFEIA5ZS7JNQ6HJSXJJDWQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.373190019 (1h29m4.412763139s) [noise=true , address=77.90.31.15:4698 , piece_id=GY2EOXKRNWWXZZMVUP4IAEMVEFKJWORCQU6K3JINKRLY5LASZK6A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.373218523 (1h29m4.412791643s) [seg_pos={0 0}, upload=1283]: Segment reached optimal threshold of 13 pieces. Sep 6 14:57:28.373338613 (1h29m4.412911733s) [noise=true , address=45.91.116.5:4297 , piece_id=QVXK6LGL2FI2YZGBEFBEC5VT2WFN4PJTZJVXZN5GMZBKRLNEL2XQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Sep 6 14:57:28.373493670 (1h29m4.41306679s) [noise=true , address=45.91.116.14:4191 , piece_id=IL6FLQPCQ6S5YJXU5VPNQEMYWZDMPJWQO6EJIQMMKN5SMO7D2JFQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Sep 6 14:57:28.373590895 (1h29m4.413164015s) [noise=true , address=77.90.31.18:4807 , piece_id=XFT4CDVMG5DSVPGT3AZK3IUXKIZAL2WAF63F6CA556QKNWZ7YPRQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection Sep 6 14:57:28.373602157 (1h29m4.413175277s) [seg_pos={0 0}, upload=1283]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 14:57:28.373664106 (1h29m4.413237236s) [upload=1283]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:57:28.373710875 (1h29m4.413283996s) [upload=1283]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:57:28.850613317 (1h29m4.890186427s) [upload=1283]: Done uploading segments... Sep 6 14:57:28.850632084 (1h29m4.890205204s) [upload=1283]: Upload finished. err: <nil> Duration: 28.023980147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [X..X.XXXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..XX] duration: 30.002301738s FAIL (2025-09-06 14:57:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:57:24.998312993 (1h29m1.037886113s) [upload=1283]: Starting upload Sep 6 14:57:24.998368709 (1h29m1.037941829s) [upload=1283]: Done starting upload Sep 6 14:57:24.998455285 (1h29m1.038028405s) [upload=1283]: Uploading segments... Sep 6 14:57:24.998541089 (1h29m1.038114209s) [upload=1283]: Got next segment. Inline: false Sep 6 14:57:24.998564915 (1h29m1.038138055s) [upload=1283]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:57:24.998569193 (1h29m1.038142313s) [upload=1283]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:57:25.859008359 (1h29m1.898581479s) [seg_pos={0 0}, upload=1283]: Begin upload segment... Sep 6 14:57:25.859054547 (1h29m1.898627667s) [seg_pos={0 0}, upload=1283]: Done begin upload segment. Sep 6 14:57:25.859073945 (1h29m1.898647055s) [noise=true , address=77.90.31.15:4699 , piece_id=TEOVTYFY5LSVQ26AUQNBRUOF6HEADWJHUJWE7DKFUGWQLERLSF5A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859078483 (1h29m1.898651603s) [noise=true , address=45.91.116.4:4940 , piece_id=2QIMH2GXC5UTNUE2FZNMUZCE3UKWNBBUAKDQHZ64UXFTF2XDWA3A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859122246 (1h29m1.898695366s) [noise=true , address=77.90.31.19:4841 , piece_id=L5STZW6EZFZ4HBNEMPVUJVK6DF6H7UQDFGDCOVDA6MUXAZH4P6EQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.859650578 (1h29m1.899223698s) [noise=true , address=77.90.31.14:4663 , piece_id=X4C654HA2AKA2M6OTHVOJDJ6OFEMAGSP6SA5PN6C26Z3KM6ER74Q, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860093936 (1h29m1.899667046s) [noise=true , address=45.91.116.7:4009 , piece_id=MHU26O5C5VLEQCD7MWUXPG7KXAMF6TSFEIA5ZS7JNQ6HJSXJJDWQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860372780 (1h29m1.8999459s) [noise=true , address=45.91.116.5:4298 , piece_id=RINQMF6FCCRXCA72TNWSMXYHFV3PQWWTQ5JRPXICONEEYZWRFHZA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860684226 (1h29m1.900257346s) [noise=true , address=45.91.116.6:4973 , piece_id=POJDDJEHVVNJSG46X654QZXXGNFTDB43CH5R4TEHLZ762V5TIENA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.860935246 (1h29m1.900508366s) [noise=true , address=45.91.116.14:4191 , piece_id=IL6FLQPCQ6S5YJXU5VPNQEMYWZDMPJWQO6EJIQMMKN5SMO7D2JFQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861147522 (1h29m1.900720642s) [noise=true , address=45.91.116.7:4010 , piece_id=2XKNJ6DQA2Y73ZIR7FQSF5Y3JQPNMIU6DGFTJ6MJZNQYZA36EGWA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861349749 (1h29m1.900922869s) [noise=true , address=45.91.116.5:4297 , piece_id=QVXK6LGL2FI2YZGBEFBEC5VT2WFN4PJTZJVXZN5GMZBKRLNEL2XQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861551104 (1h29m1.901124224s) [noise=true , address=77.90.31.15:4698 , piece_id=GY2EOXKRNWWXZZMVUP4IAEMVEFKJWORCQU6K3JINKRLY5LASZK6A, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.861826782 (1h29m1.901399902s) [noise=true , address=45.91.116.9:4262 , piece_id=IAELUYAFEFMV4F2RPPPFPBXSJ57DTESG2NTYEJXHXJBWLVNFBMSA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862073464 (1h29m1.901646574s) [noise=true , address=77.90.31.12:4592 , piece_id=ES5NTXSG57ONAFVPZNIG45GR2JDNOSJHHK7M3UIWXTQSKEQSKXFQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862278476 (1h29m1.901851596s) [noise=true , address=45.91.116.13:4153 , piece_id=JEKLLAF4VLMNYDESDVGGRN3LMWH44JK6GLHP4HR4Y7K52W5VCYTA, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.862984698 (1h29m1.902557808s) [noise=true , address=45.91.116.4:4938 , piece_id=V7EUUJ57E6ULNT6NIFKDNW42OCLIPUYXXV6RJGKGO3HOIGDZ6ZOQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:25.863506877 (1h29m1.903079997s) [noise=true , address=77.90.31.18:4807 , piece_id=XFT4CDVMG5DSVPGT3AZK3IUXKIZAL2WAF63F6CA556QKNWZ7YPRQ, seg_pos={0 0}, upload=1283]: Uploading piece... Sep 6 14:57:27.626500472 (1h29m3.666073592s) [upload=1283]: Next returned nil segment Sep 6 14:57:27.626522044 (1h29m3.666095164s) [upload=1283]: Waiting for error group managing segments... Sep 6 14:57:27.970829162 (1h29m4.010402282s) [noise=true , address=77.90.31.12:4592 , piece_id=ES5NTXSG57ONAFVPZNIG45GR2JDNOSJHHK7M3UIWXTQSKEQSKXFQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.014814389 (1h29m4.054387509s) [noise=true , address=45.91.116.9:4262 , piece_id=IAELUYAFEFMV4F2RPPPFPBXSJ57DTESG2NTYEJXHXJBWLVNFBMSA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.043386252 (1h29m4.082959362s) [noise=true , address=45.91.116.13:4153 , piece_id=JEKLLAF4VLMNYDESDVGGRN3LMWH44JK6GLHP4HR4Y7K52W5VCYTA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.052261515 (1h29m4.091834635s) [noise=true , address=77.90.31.19:4841 , piece_id=L5STZW6EZFZ4HBNEMPVUJVK6DF6H7UQDFGDCOVDA6MUXAZH4P6EQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.054781717 (1h29m4.094354838s) [noise=true , address=45.91.116.4:4940 , piece_id=2QIMH2GXC5UTNUE2FZNMUZCE3UKWNBBUAKDQHZ64UXFTF2XDWA3A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.088940631 (1h29m4.128513751s) [noise=true , address=77.90.31.14:4663 , piece_id=X4C654HA2AKA2M6OTHVOJDJ6OFEMAGSP6SA5PN6C26Z3KM6ER74Q, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.089269631 (1h29m4.128842761s) [noise=true , address=45.91.116.7:4010 , piece_id=2XKNJ6DQA2Y73ZIR7FQSF5Y3JQPNMIU6DGFTJ6MJZNQYZA36EGWA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.130989714 (1h29m4.170562834s) [noise=true , address=45.91.116.5:4298 , piece_id=RINQMF6FCCRXCA72TNWSMXYHFV3PQWWTQ5JRPXICONEEYZWRFHZA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.179333670 (1h29m4.21890679s) [noise=true , address=45.91.116.4:4938 , piece_id=V7EUUJ57E6ULNT6NIFKDNW42OCLIPUYXXV6RJGKGO3HOIGDZ6ZOQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.179384006 (1h29m4.218957116s) [noise=true , address=45.91.116.6:4973 , piece_id=POJDDJEHVVNJSG46X654QZXXGNFTDB43CH5R4TEHLZ762V5TIENA, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.183564676 (1h29m4.223137796s) [noise=true , address=77.90.31.15:4699 , piece_id=TEOVTYFY5LSVQ26AUQNBRUOF6HEADWJHUJWE7DKFUGWQLERLSF5A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.220931549 (1h29m4.260504669s) [noise=true , address=45.91.116.7:4009 , piece_id=MHU26O5C5VLEQCD7MWUXPG7KXAMF6TSFEIA5ZS7JNQ6HJSXJJDWQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.373190019 (1h29m4.412763139s) [noise=true , address=77.90.31.15:4698 , piece_id=GY2EOXKRNWWXZZMVUP4IAEMVEFKJWORCQU6K3JINKRLY5LASZK6A, seg_pos={0 0}, upload=1283]: Done uploading piece. err: <nil> Sep 6 14:57:28.373218523 (1h29m4.412791643s) [seg_pos={0 0}, upload=1283]: Segment reached optimal threshold of 13 pieces. Sep 6 14:57:28.373338613 (1h29m4.412911733s) [noise=true , address=45.91.116.5:4297 , piece_id=QVXK6LGL2FI2YZGBEFBEC5VT2WFN4PJTZJVXZN5GMZBKRLNEL2XQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Sep 6 14:57:28.373493670 (1h29m4.41306679s) [noise=true , address=45.91.116.14:4191 , piece_id=IL6FLQPCQ6S5YJXU5VPNQEMYWZDMPJWQO6EJIQMMKN5SMO7D2JFQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Sep 6 14:57:28.373590895 (1h29m4.413164015s) [noise=true , address=77.90.31.18:4807 , piece_id=XFT4CDVMG5DSVPGT3AZK3IUXKIZAL2WAF63F6CA556QKNWZ7YPRQ, seg_pos={0 0}, upload=1283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52844->77.90.31.18:4807: use of closed network connection Sep 6 14:57:28.373602157 (1h29m4.413175277s) [seg_pos={0 0}, upload=1283]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 14:57:28.373664106 (1h29m4.413237236s) [upload=1283]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:57:28.373710875 (1h29m4.413283996s) [upload=1283]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:57:28.850613317 (1h29m4.890186427s) [upload=1283]: Done uploading segments... Sep 6 14:57:28.850632084 (1h29m4.890205204s) [upload=1283]: Upload finished. err: <nil> Duration: 30.002325735s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.12538427s FAIL (2025-09-06 14:57:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.125413486s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 35.126958784s FAIL (2025-09-06 14:57:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 35.127012738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.X..X.XXXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..X] duration: 30.002568056s FAIL (2025-09-06 14:57:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:56:47.885321914 (1h28m23.924895034s) [upload=1274]: Starting upload Sep 6 14:56:47.885375436 (1h28m23.924948556s) [upload=1274]: Done starting upload Sep 6 14:56:47.885500807 (1h28m23.925073927s) [upload=1274]: Uploading segments... Sep 6 14:56:47.885592812 (1h28m23.925165922s) [upload=1274]: Got next segment. Inline: false Sep 6 14:56:47.885634012 (1h28m23.925207132s) [upload=1274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:56:47.885639552 (1h28m23.925212672s) [upload=1274]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:56:48.158372742 (1h28m24.197945863s) [seg_pos={0 0}, upload=1274]: Begin upload segment... Sep 6 14:56:48.158426766 (1h28m24.197999886s) [seg_pos={0 0}, upload=1274]: Done begin upload segment. Sep 6 14:56:48.158452946 (1h28m24.198026066s) [noise=true , address=45.91.116.13:4154 , piece_id=Z4YPAAMN7GT5YTW47PXF5XILGPWW5RQJ4L442HK6OPOY4C245ZIA, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.158483364 (1h28m24.198056484s) [noise=true , address=45.91.116.6:4974 , piece_id=DR3WEL2QER25GMEJ6EH2STR4MB4KEKEF4MYMW2SLE4WLK46V6FCA, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.158897677 (1h28m24.198470797s) [noise=true , address=77.90.31.15:4699 , piece_id=WGD72DIFA2OTL74CTJHCY5BRAVBV5TQE3AUKQ4PXZKBGXL44JSNQ, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.158908167 (1h28m24.198481277s) [noise=true , address=45.91.116.9:4262 , piece_id=7OAFX4J3MUMK5UVTIEMSS7YNFVLOZGKUOWSJ4PD4NYJAARNI3I4A, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.159171962 (1h28m24.198745082s) [noise=true , address=77.90.31.10:4517 , piece_id=CBBKG2TJQGDLZ4NTST64C6SAQK5XCKLU26WAE5NYHV5GFVBIQJKQ, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.159211558 (1h28m24.198784678s) [noise=true , address=45.91.116.6:4973 , piece_id=ASVQ4FHGYSZUP3XADR25SLNW3RKMAF5EFPMUBCFGM276PJLPXQEA, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.159507715 (1h28m24.199080835s) [noise=true , address=77.90.31.10:4518 , piece_id=4B5477D6HEE3D4K7VYKBPTVHXVLZLUAZ6RXEW5HE7WL62OHIF2VA, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.159791547 (1h28m24.199364668s) [noise=true , address=45.91.116.7:4011 , piece_id=TPCJT2VHYPH5IJZ73WPVA6YH3VFN4TGW7G2MMSFBQEOY4ELXH4PA, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.160029934 (1h28m24.199603054s) [noise=true , address=77.90.31.13:4626 , piece_id=6KJ7HQBV4PQWFITQHEWGJ6DDIVDQR7J4A6Z2J6YWOI6XRVG5DKVA, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.160267167 (1h28m24.199840287s) [noise=true , address=77.90.31.14:4663 , piece_id=QXFPWP36YI3B4LMETIZQ2WX3AO5VKEZAGWKR326KVAXCAAN4BRNA, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.160541592 (1h28m24.200114713s) [noise=true , address=45.91.116.5:4299 , piece_id=6XDJQDEIKU6CQBRX7T726VPUPFIDEEA6C2QBMSMTIV42YU3TQ7ZQ, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.160763127 (1h28m24.200336247s) [noise=true , address=77.90.31.21:4878 , piece_id=7XSXJIYDYYBE6Q5FALIALMYWE7GCSUEH36WWJ4DX3YZI6HYKN72Q, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.160968089 (1h28m24.200541209s) [noise=true , address=45.91.116.11:4119 , piece_id=5SUOS4532IHLR76LYYTBOERBYG53V4SXFUUJ3VXO2AAXJNXGCLZQ, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.161180896 (1h28m24.200754016s) [noise=true , address=45.91.116.10:4081 , piece_id=EIT356ZSRTSDOKR47SLJ4PILXFDNTU7AQGF7RNESKFJ5KSQLO4CA, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.161380638 (1h28m24.200953758s) [noise=true , address=45.91.116.4:4937 , piece_id=EDJWA5M5H3L3XPWHDLYLMQPWKPCMIS2QYIGIV6PG5HAP5OEZD7NA, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:48.161596291 (1h28m24.201169411s) [noise=true , address=45.91.116.6:4975 , piece_id=O7FKM4HFQ2JGVZ5RMA3PFCYZPGQRW5MN6Y7X6U3TQGKLADTCXFSQ, seg_pos={0 0}, upload=1274]: Uploading piece... Sep 6 14:56:50.027947091 (1h28m26.067520211s) [upload=1274]: Next returned nil segment Sep 6 14:56:50.027965286 (1h28m26.067538407s) [upload=1274]: Waiting for error group managing segments... Sep 6 14:56:50.358985250 (1h28m26.39855836s) [noise=true , address=45.91.116.7:4011 , piece_id=TPCJT2VHYPH5IJZ73WPVA6YH3VFN4TGW7G2MMSFBQEOY4ELXH4PA, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:50.376021910 (1h28m26.41559503s) [noise=true , address=45.91.116.4:4937 , piece_id=EDJWA5M5H3L3XPWHDLYLMQPWKPCMIS2QYIGIV6PG5HAP5OEZD7NA, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:50.393399011 (1h28m26.432972131s) [noise=true , address=45.91.116.9:4262 , piece_id=7OAFX4J3MUMK5UVTIEMSS7YNFVLOZGKUOWSJ4PD4NYJAARNI3I4A, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:50.408042312 (1h28m26.447615432s) [noise=true , address=45.91.116.5:4299 , piece_id=6XDJQDEIKU6CQBRX7T726VPUPFIDEEA6C2QBMSMTIV42YU3TQ7ZQ, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:50.424241470 (1h28m26.46381459s) [noise=true , address=45.91.116.11:4119 , piece_id=5SUOS4532IHLR76LYYTBOERBYG53V4SXFUUJ3VXO2AAXJNXGCLZQ, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:50.429840764 (1h28m26.469413884s) [noise=true , address=77.90.31.10:4517 , piece_id=CBBKG2TJQGDLZ4NTST64C6SAQK5XCKLU26WAE5NYHV5GFVBIQJKQ, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:50.443224143 (1h28m26.482797263s) [noise=true , address=45.91.116.6:4975 , piece_id=O7FKM4HFQ2JGVZ5RMA3PFCYZPGQRW5MN6Y7X6U3TQGKLADTCXFSQ, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:50.468232277 (1h28m26.507805397s) [noise=true , address=45.91.116.6:4974 , piece_id=DR3WEL2QER25GMEJ6EH2STR4MB4KEKEF4MYMW2SLE4WLK46V6FCA, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:50.499519415 (1h28m26.539092535s) [noise=true , address=77.90.31.10:4518 , piece_id=4B5477D6HEE3D4K7VYKBPTVHXVLZLUAZ6RXEW5HE7WL62OHIF2VA, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:50.505472026 (1h28m26.545045146s) [noise=true , address=77.90.31.21:4878 , piece_id=7XSXJIYDYYBE6Q5FALIALMYWE7GCSUEH36WWJ4DX3YZI6HYKN72Q, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:50.537545269 (1h28m26.577118389s) [noise=true , address=77.90.31.13:4626 , piece_id=6KJ7HQBV4PQWFITQHEWGJ6DDIVDQR7J4A6Z2J6YWOI6XRVG5DKVA, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:50.582816696 (1h28m26.622389806s) [noise=true , address=45.91.116.10:4081 , piece_id=EIT356ZSRTSDOKR47SLJ4PILXFDNTU7AQGF7RNESKFJ5KSQLO4CA, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:51.270247049 (1h28m27.309820169s) [noise=true , address=77.90.31.14:4663 , piece_id=QXFPWP36YI3B4LMETIZQ2WX3AO5VKEZAGWKR326KVAXCAAN4BRNA, seg_pos={0 0}, upload=1274]: Done uploading piece. err: <nil> Sep 6 14:56:51.270274701 (1h28m27.309847821s) [seg_pos={0 0}, upload=1274]: Segment reached optimal threshold of 13 pieces. Sep 6 14:56:51.270385974 (1h28m27.309959094s) [noise=true , address=77.90.31.15:4699 , piece_id=WGD72DIFA2OTL74CTJHCY5BRAVBV5TQE3AUKQ4PXZKBGXL44JSNQ, seg_pos={0 0}, upload=1274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 6 14:56:51.270438575 (1h28m27.310011705s) [noise=true , address=45.91.116.13:4154 , piece_id=Z4YPAAMN7GT5YTW47PXF5XILGPWW5RQJ4L442HK6OPOY4C245ZIA, seg_pos={0 0}, upload=1274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Sep 6 14:56:51.270515653 (1h28m27.310088773s) [noise=true , address=45.91.116.6:4973 , piece_id=ASVQ4FHGYSZUP3XADR25SLNW3RKMAF5EFPMUBCFGM276PJLPXQEA, seg_pos={0 0}, upload=1274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:42808->45.91.116.6:4973: use of closed network connection; conn closed: op Sep 6 14:56:51.270534800 (1h28m27.31010792s) [seg_pos={0 0}, upload=1274]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 14:56:51.270604462 (1h28m27.310177582s) [upload=1274]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:56:51.270608760 (1h28m27.31018188s) [upload=1274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:56:52.374937377 (1h28m28.414510487s) [upload=1274]: Done uploading segments... Sep 6 14:56:52.374954229 (1h28m28.414527339s) [upload=1274]: Upload finished. err: <nil> Duration: 30.002598264s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.788056958s FAIL (2025-09-06 14:57:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.788078609s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.984192849s FAIL (2025-09-06 14:56:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.984210493s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [..X..X.XXXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X..] duration: 21.871108057s FAIL (2025-09-06 14:56:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 6 14:54:27.689941286 (1h26m3.729514406s) [upload=1243]: Starting upload Sep 6 14:54:27.689998335 (1h26m3.729571445s) [upload=1243]: Done starting upload Sep 6 14:54:27.690106863 (1h26m3.729679973s) [upload=1243]: Uploading segments... Sep 6 14:54:27.690190273 (1h26m3.729763393s) [upload=1243]: Got next segment. Inline: false Sep 6 14:54:27.690246059 (1h26m3.729819219s) [upload=1243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:54:27.690256831 (1h26m3.729830021s) [upload=1243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:54:28.080361644 (1h26m4.119934815s) [seg_pos={0 0}, upload=1243]: Begin upload segment... Sep 6 14:54:28.080429243 (1h26m4.120002383s) [seg_pos={0 0}, upload=1243]: Done begin upload segment. Sep 6 14:54:28.080454011 (1h26m4.120027171s) [noise=true , address=77.90.31.14:4663 , piece_id=7LKUXEKFYCUE4W747U2XU7B43WYLBKG4YPW5R5C553XWG2EU4MMA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.080463238 (1h26m4.120036358s) [noise=true , address=45.91.116.4:4938 , piece_id=QE36UA7RK3KMRWJIRO4X7FXIF6VJYCZABXJB3CQJR4SYZVMKAUIA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.080541117 (1h26m4.120114257s) [noise=true , address=45.91.116.10:4081 , piece_id=5ZDZSMH3IGQJQ3GJJZR4S63VH45C443Z3FJ3I6BPD43DDUQ2GOIQ, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.081223683 (1h26m4.120796803s) [noise=true , address=45.91.116.13:4155 , piece_id=24BHRE2TNUQH4HFYIORZTLHDXUI3HIJYIRGTKQURDVNRVHXLCKXA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.081691919 (1h26m4.121265059s) [noise=true , address=45.91.116.13:4153 , piece_id=LKUUWCFJMPPLNWSVQY4XOFVKKBA3574H7D234DE4ANZOSOMX4QCA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.082169243 (1h26m4.121742363s) [noise=true , address=45.91.116.11:4117 , piece_id=YACV7DNCNV4AMB7H6GUHKFRTH2K7XQAJDKNCK7ASZZGBJX5JIRNA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.082595158 (1h26m4.122168288s) [noise=true , address=45.91.116.8:4048 , piece_id=4K4OWANLKT45OKCCWKTC63OW53NM55GGXMVNMUYUZLNCX7L7YK4A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.083010593 (1h26m4.122583723s) [noise=true , address=45.91.116.6:4974 , piece_id=YUC2L3NW6GL5XXZO2ABDKDKM2VG3UJBP2LRBLSJZDNVHMK6POQ7A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.083370882 (1h26m4.122944013s) [noise=true , address=45.91.116.11:4118 , piece_id=YPSDBARV75LMOHLJYOU54CV3B2I242IRO2D6U6PIYAEHCE2OOR6Q, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.083782580 (1h26m4.1233557s) [noise=true , address=45.91.116.4:4937 , piece_id=24SIDYNY4GAPOXFKE5M3YWPRCKS4COYABJFX55TYE6THDDOYO7VQ, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.084149171 (1h26m4.123722291s) [noise=true , address=77.90.31.12:4590 , piece_id=MKQ4GL6XDA6XD55HDZ574RSDH4SUJ7PXILO3SYCFA5XFW3UHV55A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.084492559 (1h26m4.124065679s) [noise=true , address=77.90.31.15:4698 , piece_id=V4MHEW4F6D22RSTB4SXM2PQF6MG4IGUHIZSFPVIW4XYJZXDI35WA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.084823893 (1h26m4.124397023s) [noise=true , address=45.91.116.14:4191 , piece_id=UBFUDYK4AWPGQSJGAHMSQQQHQVAZLXFW57MNZVHCJC6IV64PKKAQ, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.085102225 (1h26m4.124675345s) [noise=true , address=77.90.31.15:4699 , piece_id=IZDH4EVKQHBCQMBRTNW4LA55HCLZXKB3B64DJDZ4RPEFKQEW6TCA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.085421847 (1h26m4.124994967s) [noise=true , address=45.91.116.5:4299 , piece_id=JT5Q6AR4XDPABXUP2MWQYPWCPXLIE2DHZD2FKQKTKMAMEJKFVDVA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.085715930 (1h26m4.12528905s) [noise=true , address=45.91.116.5:4297 , piece_id=GJ56CLX5HTB4LAXL3QE4234WWUJXANB4N3SUIRVMULNBWTXW5L6A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:31.987150555 (1h26m8.026723675s) [upload=1243]: Next returned nil segment Sep 6 14:54:31.987168620 (1h26m8.02674174s) [upload=1243]: Waiting for error group managing segments... Sep 6 14:54:32.751850966 (1h26m8.791424086s) [noise=true , address=45.91.116.14:4191 , piece_id=UBFUDYK4AWPGQSJGAHMSQQQHQVAZLXFW57MNZVHCJC6IV64PKKAQ, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:33.198991208 (1h26m9.238564328s) [noise=true , address=45.91.116.13:4155 , piece_id=24BHRE2TNUQH4HFYIORZTLHDXUI3HIJYIRGTKQURDVNRVHXLCKXA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:33.658165727 (1h26m9.697738837s) [noise=true , address=77.90.31.15:4698 , piece_id=V4MHEW4F6D22RSTB4SXM2PQF6MG4IGUHIZSFPVIW4XYJZXDI35WA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:33.998717471 (1h26m10.038290591s) [noise=true , address=45.91.116.6:4974 , piece_id=YUC2L3NW6GL5XXZO2ABDKDKM2VG3UJBP2LRBLSJZDNVHMK6POQ7A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.030174785 (1h26m10.069747905s) [noise=true , address=45.91.116.11:4118 , piece_id=YPSDBARV75LMOHLJYOU54CV3B2I242IRO2D6U6PIYAEHCE2OOR6Q, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.361124914 (1h26m10.400698084s) [noise=true , address=77.90.31.14:4663 , piece_id=7LKUXEKFYCUE4W747U2XU7B43WYLBKG4YPW5R5C553XWG2EU4MMA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.416770714 (1h26m10.456343834s) [noise=true , address=77.90.31.12:4590 , piece_id=MKQ4GL6XDA6XD55HDZ574RSDH4SUJ7PXILO3SYCFA5XFW3UHV55A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.418617908 (1h26m10.458191028s) [noise=true , address=45.91.116.5:4299 , piece_id=JT5Q6AR4XDPABXUP2MWQYPWCPXLIE2DHZD2FKQKTKMAMEJKFVDVA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.479957664 (1h26m10.519530774s) [noise=true , address=45.91.116.11:4117 , piece_id=YACV7DNCNV4AMB7H6GUHKFRTH2K7XQAJDKNCK7ASZZGBJX5JIRNA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.864602039 (1h26m10.904175159s) [noise=true , address=45.91.116.10:4081 , piece_id=5ZDZSMH3IGQJQ3GJJZR4S63VH45C443Z3FJ3I6BPD43DDUQ2GOIQ, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:35.127421269 (1h26m11.166994389s) [noise=true , address=45.91.116.8:4048 , piece_id=4K4OWANLKT45OKCCWKTC63OW53NM55GGXMVNMUYUZLNCX7L7YK4A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:35.750575627 (1h26m11.790148747s) [noise=true , address=45.91.116.4:4938 , piece_id=QE36UA7RK3KMRWJIRO4X7FXIF6VJYCZABXJB3CQJR4SYZVMKAUIA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:36.621200325 (1h26m12.660773456s) [noise=true , address=77.90.31.15:4699 , piece_id=IZDH4EVKQHBCQMBRTNW4LA55HCLZXKB3B64DJDZ4RPEFKQEW6TCA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:36.621233669 (1h26m12.660806789s) [seg_pos={0 0}, upload=1243]: Segment reached optimal threshold of 13 pieces. Sep 6 14:54:36.621458038 (1h26m12.661031168s) [noise=true , address=45.91.116.4:4937 , piece_id=24SIDYNY4GAPOXFKE5M3YWPRCKS4COYABJFX55TYE6THDDOYO7VQ, seg_pos={0 0}, upload=1243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:36.621566305 (1h26m12.661139435s) [noise=true , address=45.91.116.5:4297 , piece_id=GJ56CLX5HTB4LAXL3QE4234WWUJXANB4N3SUIRVMULNBWTXW5L6A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:50752->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:50752->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50752->45.91.116.5:4297: use of closed network connection Sep 6 14:54:36.621588298 (1h26m12.661161418s) [noise=true , address=45.91.116.13:4153 , piece_id=LKUUWCFJMPPLNWSVQY4XOFVKKBA3574H7D234DE4ANZOSOMX4QCA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:48492->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:48492->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48492->45.91.116.13:4153: use of closed network connection Sep 6 14:54:36.621640548 (1h26m12.661213678s) [seg_pos={0 0}, upload=1243]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 14:54:36.621760889 (1h26m12.661334009s) [upload=1243]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:36.621769505 (1h26m12.661342625s) [upload=1243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:37.082190940 (1h26m13.12176405s) [upload=1243]: Done uploading segments... Sep 6 14:54:37.082210087 (1h26m13.121783207s) [upload=1243]: Upload finished. err: <nil> Duration: 21.87113588s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.590493445s FAIL (2025-09-06 14:56:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.590507662s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 14.944234794s FAIL (2025-09-06 14:56:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 14.944271434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 7.038928736s FAIL (2025-09-06 14:55:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.038948163s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 27.242981922s FAIL (2025-09-06 14:55:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 27.242997743s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [...X..X.XXXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X.] duration: 9.652545155s FAIL (2025-09-06 14:55:06 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 6 14:54:27.689941286 (1h26m3.729514406s) [upload=1243]: Starting upload Sep 6 14:54:27.689998335 (1h26m3.729571445s) [upload=1243]: Done starting upload Sep 6 14:54:27.690106863 (1h26m3.729679973s) [upload=1243]: Uploading segments... Sep 6 14:54:27.690190273 (1h26m3.729763393s) [upload=1243]: Got next segment. Inline: false Sep 6 14:54:27.690246059 (1h26m3.729819219s) [upload=1243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:54:27.690256831 (1h26m3.729830021s) [upload=1243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:54:28.080361644 (1h26m4.119934815s) [seg_pos={0 0}, upload=1243]: Begin upload segment... Sep 6 14:54:28.080429243 (1h26m4.120002383s) [seg_pos={0 0}, upload=1243]: Done begin upload segment. Sep 6 14:54:28.080454011 (1h26m4.120027171s) [noise=true , address=77.90.31.14:4663 , piece_id=7LKUXEKFYCUE4W747U2XU7B43WYLBKG4YPW5R5C553XWG2EU4MMA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.080463238 (1h26m4.120036358s) [noise=true , address=45.91.116.4:4938 , piece_id=QE36UA7RK3KMRWJIRO4X7FXIF6VJYCZABXJB3CQJR4SYZVMKAUIA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.080541117 (1h26m4.120114257s) [noise=true , address=45.91.116.10:4081 , piece_id=5ZDZSMH3IGQJQ3GJJZR4S63VH45C443Z3FJ3I6BPD43DDUQ2GOIQ, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.081223683 (1h26m4.120796803s) [noise=true , address=45.91.116.13:4155 , piece_id=24BHRE2TNUQH4HFYIORZTLHDXUI3HIJYIRGTKQURDVNRVHXLCKXA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.081691919 (1h26m4.121265059s) [noise=true , address=45.91.116.13:4153 , piece_id=LKUUWCFJMPPLNWSVQY4XOFVKKBA3574H7D234DE4ANZOSOMX4QCA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.082169243 (1h26m4.121742363s) [noise=true , address=45.91.116.11:4117 , piece_id=YACV7DNCNV4AMB7H6GUHKFRTH2K7XQAJDKNCK7ASZZGBJX5JIRNA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.082595158 (1h26m4.122168288s) [noise=true , address=45.91.116.8:4048 , piece_id=4K4OWANLKT45OKCCWKTC63OW53NM55GGXMVNMUYUZLNCX7L7YK4A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.083010593 (1h26m4.122583723s) [noise=true , address=45.91.116.6:4974 , piece_id=YUC2L3NW6GL5XXZO2ABDKDKM2VG3UJBP2LRBLSJZDNVHMK6POQ7A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.083370882 (1h26m4.122944013s) [noise=true , address=45.91.116.11:4118 , piece_id=YPSDBARV75LMOHLJYOU54CV3B2I242IRO2D6U6PIYAEHCE2OOR6Q, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.083782580 (1h26m4.1233557s) [noise=true , address=45.91.116.4:4937 , piece_id=24SIDYNY4GAPOXFKE5M3YWPRCKS4COYABJFX55TYE6THDDOYO7VQ, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.084149171 (1h26m4.123722291s) [noise=true , address=77.90.31.12:4590 , piece_id=MKQ4GL6XDA6XD55HDZ574RSDH4SUJ7PXILO3SYCFA5XFW3UHV55A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.084492559 (1h26m4.124065679s) [noise=true , address=77.90.31.15:4698 , piece_id=V4MHEW4F6D22RSTB4SXM2PQF6MG4IGUHIZSFPVIW4XYJZXDI35WA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.084823893 (1h26m4.124397023s) [noise=true , address=45.91.116.14:4191 , piece_id=UBFUDYK4AWPGQSJGAHMSQQQHQVAZLXFW57MNZVHCJC6IV64PKKAQ, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.085102225 (1h26m4.124675345s) [noise=true , address=77.90.31.15:4699 , piece_id=IZDH4EVKQHBCQMBRTNW4LA55HCLZXKB3B64DJDZ4RPEFKQEW6TCA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.085421847 (1h26m4.124994967s) [noise=true , address=45.91.116.5:4299 , piece_id=JT5Q6AR4XDPABXUP2MWQYPWCPXLIE2DHZD2FKQKTKMAMEJKFVDVA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.085715930 (1h26m4.12528905s) [noise=true , address=45.91.116.5:4297 , piece_id=GJ56CLX5HTB4LAXL3QE4234WWUJXANB4N3SUIRVMULNBWTXW5L6A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:31.987150555 (1h26m8.026723675s) [upload=1243]: Next returned nil segment Sep 6 14:54:31.987168620 (1h26m8.02674174s) [upload=1243]: Waiting for error group managing segments... Sep 6 14:54:32.751850966 (1h26m8.791424086s) [noise=true , address=45.91.116.14:4191 , piece_id=UBFUDYK4AWPGQSJGAHMSQQQHQVAZLXFW57MNZVHCJC6IV64PKKAQ, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:33.198991208 (1h26m9.238564328s) [noise=true , address=45.91.116.13:4155 , piece_id=24BHRE2TNUQH4HFYIORZTLHDXUI3HIJYIRGTKQURDVNRVHXLCKXA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:33.658165727 (1h26m9.697738837s) [noise=true , address=77.90.31.15:4698 , piece_id=V4MHEW4F6D22RSTB4SXM2PQF6MG4IGUHIZSFPVIW4XYJZXDI35WA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:33.998717471 (1h26m10.038290591s) [noise=true , address=45.91.116.6:4974 , piece_id=YUC2L3NW6GL5XXZO2ABDKDKM2VG3UJBP2LRBLSJZDNVHMK6POQ7A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.030174785 (1h26m10.069747905s) [noise=true , address=45.91.116.11:4118 , piece_id=YPSDBARV75LMOHLJYOU54CV3B2I242IRO2D6U6PIYAEHCE2OOR6Q, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.361124914 (1h26m10.400698084s) [noise=true , address=77.90.31.14:4663 , piece_id=7LKUXEKFYCUE4W747U2XU7B43WYLBKG4YPW5R5C553XWG2EU4MMA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.416770714 (1h26m10.456343834s) [noise=true , address=77.90.31.12:4590 , piece_id=MKQ4GL6XDA6XD55HDZ574RSDH4SUJ7PXILO3SYCFA5XFW3UHV55A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.418617908 (1h26m10.458191028s) [noise=true , address=45.91.116.5:4299 , piece_id=JT5Q6AR4XDPABXUP2MWQYPWCPXLIE2DHZD2FKQKTKMAMEJKFVDVA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.479957664 (1h26m10.519530774s) [noise=true , address=45.91.116.11:4117 , piece_id=YACV7DNCNV4AMB7H6GUHKFRTH2K7XQAJDKNCK7ASZZGBJX5JIRNA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.864602039 (1h26m10.904175159s) [noise=true , address=45.91.116.10:4081 , piece_id=5ZDZSMH3IGQJQ3GJJZR4S63VH45C443Z3FJ3I6BPD43DDUQ2GOIQ, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:35.127421269 (1h26m11.166994389s) [noise=true , address=45.91.116.8:4048 , piece_id=4K4OWANLKT45OKCCWKTC63OW53NM55GGXMVNMUYUZLNCX7L7YK4A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:35.750575627 (1h26m11.790148747s) [noise=true , address=45.91.116.4:4938 , piece_id=QE36UA7RK3KMRWJIRO4X7FXIF6VJYCZABXJB3CQJR4SYZVMKAUIA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:36.621200325 (1h26m12.660773456s) [noise=true , address=77.90.31.15:4699 , piece_id=IZDH4EVKQHBCQMBRTNW4LA55HCLZXKB3B64DJDZ4RPEFKQEW6TCA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:36.621233669 (1h26m12.660806789s) [seg_pos={0 0}, upload=1243]: Segment reached optimal threshold of 13 pieces. Sep 6 14:54:36.621458038 (1h26m12.661031168s) [noise=true , address=45.91.116.4:4937 , piece_id=24SIDYNY4GAPOXFKE5M3YWPRCKS4COYABJFX55TYE6THDDOYO7VQ, seg_pos={0 0}, upload=1243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:36.621566305 (1h26m12.661139435s) [noise=true , address=45.91.116.5:4297 , piece_id=GJ56CLX5HTB4LAXL3QE4234WWUJXANB4N3SUIRVMULNBWTXW5L6A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:50752->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:50752->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50752->45.91.116.5:4297: use of closed network connection Sep 6 14:54:36.621588298 (1h26m12.661161418s) [noise=true , address=45.91.116.13:4153 , piece_id=LKUUWCFJMPPLNWSVQY4XOFVKKBA3574H7D234DE4ANZOSOMX4QCA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:48492->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:48492->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48492->45.91.116.13:4153: use of closed network connection Sep 6 14:54:36.621640548 (1h26m12.661213678s) [seg_pos={0 0}, upload=1243]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 14:54:36.621760889 (1h26m12.661334009s) [upload=1243]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:36.621769505 (1h26m12.661342625s) [upload=1243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:37.082190940 (1h26m13.12176405s) [upload=1243]: Done uploading segments... Sep 6 14:54:37.082210087 (1h26m13.121783207s) [upload=1243]: Upload finished. err: <nil> Duration: 9.65256312s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.843414125s FAIL (2025-09-06 14:54:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.84343244s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [....X..X.XXXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...X] duration: 30.001753566s FAIL (2025-09-06 14:54:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 6 14:54:27.689941286 (1h26m3.729514406s) [upload=1243]: Starting upload Sep 6 14:54:27.689998335 (1h26m3.729571445s) [upload=1243]: Done starting upload Sep 6 14:54:27.690106863 (1h26m3.729679973s) [upload=1243]: Uploading segments... Sep 6 14:54:27.690190273 (1h26m3.729763393s) [upload=1243]: Got next segment. Inline: false Sep 6 14:54:27.690246059 (1h26m3.729819219s) [upload=1243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:54:27.690256831 (1h26m3.729830021s) [upload=1243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:54:28.080361644 (1h26m4.119934815s) [seg_pos={0 0}, upload=1243]: Begin upload segment... Sep 6 14:54:28.080429243 (1h26m4.120002383s) [seg_pos={0 0}, upload=1243]: Done begin upload segment. Sep 6 14:54:28.080454011 (1h26m4.120027171s) [noise=true , address=77.90.31.14:4663 , piece_id=7LKUXEKFYCUE4W747U2XU7B43WYLBKG4YPW5R5C553XWG2EU4MMA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.080463238 (1h26m4.120036358s) [noise=true , address=45.91.116.4:4938 , piece_id=QE36UA7RK3KMRWJIRO4X7FXIF6VJYCZABXJB3CQJR4SYZVMKAUIA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.080541117 (1h26m4.120114257s) [noise=true , address=45.91.116.10:4081 , piece_id=5ZDZSMH3IGQJQ3GJJZR4S63VH45C443Z3FJ3I6BPD43DDUQ2GOIQ, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.081223683 (1h26m4.120796803s) [noise=true , address=45.91.116.13:4155 , piece_id=24BHRE2TNUQH4HFYIORZTLHDXUI3HIJYIRGTKQURDVNRVHXLCKXA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.081691919 (1h26m4.121265059s) [noise=true , address=45.91.116.13:4153 , piece_id=LKUUWCFJMPPLNWSVQY4XOFVKKBA3574H7D234DE4ANZOSOMX4QCA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.082169243 (1h26m4.121742363s) [noise=true , address=45.91.116.11:4117 , piece_id=YACV7DNCNV4AMB7H6GUHKFRTH2K7XQAJDKNCK7ASZZGBJX5JIRNA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.082595158 (1h26m4.122168288s) [noise=true , address=45.91.116.8:4048 , piece_id=4K4OWANLKT45OKCCWKTC63OW53NM55GGXMVNMUYUZLNCX7L7YK4A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.083010593 (1h26m4.122583723s) [noise=true , address=45.91.116.6:4974 , piece_id=YUC2L3NW6GL5XXZO2ABDKDKM2VG3UJBP2LRBLSJZDNVHMK6POQ7A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.083370882 (1h26m4.122944013s) [noise=true , address=45.91.116.11:4118 , piece_id=YPSDBARV75LMOHLJYOU54CV3B2I242IRO2D6U6PIYAEHCE2OOR6Q, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.083782580 (1h26m4.1233557s) [noise=true , address=45.91.116.4:4937 , piece_id=24SIDYNY4GAPOXFKE5M3YWPRCKS4COYABJFX55TYE6THDDOYO7VQ, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.084149171 (1h26m4.123722291s) [noise=true , address=77.90.31.12:4590 , piece_id=MKQ4GL6XDA6XD55HDZ574RSDH4SUJ7PXILO3SYCFA5XFW3UHV55A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.084492559 (1h26m4.124065679s) [noise=true , address=77.90.31.15:4698 , piece_id=V4MHEW4F6D22RSTB4SXM2PQF6MG4IGUHIZSFPVIW4XYJZXDI35WA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.084823893 (1h26m4.124397023s) [noise=true , address=45.91.116.14:4191 , piece_id=UBFUDYK4AWPGQSJGAHMSQQQHQVAZLXFW57MNZVHCJC6IV64PKKAQ, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.085102225 (1h26m4.124675345s) [noise=true , address=77.90.31.15:4699 , piece_id=IZDH4EVKQHBCQMBRTNW4LA55HCLZXKB3B64DJDZ4RPEFKQEW6TCA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.085421847 (1h26m4.124994967s) [noise=true , address=45.91.116.5:4299 , piece_id=JT5Q6AR4XDPABXUP2MWQYPWCPXLIE2DHZD2FKQKTKMAMEJKFVDVA, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:28.085715930 (1h26m4.12528905s) [noise=true , address=45.91.116.5:4297 , piece_id=GJ56CLX5HTB4LAXL3QE4234WWUJXANB4N3SUIRVMULNBWTXW5L6A, seg_pos={0 0}, upload=1243]: Uploading piece... Sep 6 14:54:31.987150555 (1h26m8.026723675s) [upload=1243]: Next returned nil segment Sep 6 14:54:31.987168620 (1h26m8.02674174s) [upload=1243]: Waiting for error group managing segments... Sep 6 14:54:32.751850966 (1h26m8.791424086s) [noise=true , address=45.91.116.14:4191 , piece_id=UBFUDYK4AWPGQSJGAHMSQQQHQVAZLXFW57MNZVHCJC6IV64PKKAQ, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:33.198991208 (1h26m9.238564328s) [noise=true , address=45.91.116.13:4155 , piece_id=24BHRE2TNUQH4HFYIORZTLHDXUI3HIJYIRGTKQURDVNRVHXLCKXA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:33.658165727 (1h26m9.697738837s) [noise=true , address=77.90.31.15:4698 , piece_id=V4MHEW4F6D22RSTB4SXM2PQF6MG4IGUHIZSFPVIW4XYJZXDI35WA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:33.998717471 (1h26m10.038290591s) [noise=true , address=45.91.116.6:4974 , piece_id=YUC2L3NW6GL5XXZO2ABDKDKM2VG3UJBP2LRBLSJZDNVHMK6POQ7A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.030174785 (1h26m10.069747905s) [noise=true , address=45.91.116.11:4118 , piece_id=YPSDBARV75LMOHLJYOU54CV3B2I242IRO2D6U6PIYAEHCE2OOR6Q, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.361124914 (1h26m10.400698084s) [noise=true , address=77.90.31.14:4663 , piece_id=7LKUXEKFYCUE4W747U2XU7B43WYLBKG4YPW5R5C553XWG2EU4MMA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.416770714 (1h26m10.456343834s) [noise=true , address=77.90.31.12:4590 , piece_id=MKQ4GL6XDA6XD55HDZ574RSDH4SUJ7PXILO3SYCFA5XFW3UHV55A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.418617908 (1h26m10.458191028s) [noise=true , address=45.91.116.5:4299 , piece_id=JT5Q6AR4XDPABXUP2MWQYPWCPXLIE2DHZD2FKQKTKMAMEJKFVDVA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.479957664 (1h26m10.519530774s) [noise=true , address=45.91.116.11:4117 , piece_id=YACV7DNCNV4AMB7H6GUHKFRTH2K7XQAJDKNCK7ASZZGBJX5JIRNA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:34.864602039 (1h26m10.904175159s) [noise=true , address=45.91.116.10:4081 , piece_id=5ZDZSMH3IGQJQ3GJJZR4S63VH45C443Z3FJ3I6BPD43DDUQ2GOIQ, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:35.127421269 (1h26m11.166994389s) [noise=true , address=45.91.116.8:4048 , piece_id=4K4OWANLKT45OKCCWKTC63OW53NM55GGXMVNMUYUZLNCX7L7YK4A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:35.750575627 (1h26m11.790148747s) [noise=true , address=45.91.116.4:4938 , piece_id=QE36UA7RK3KMRWJIRO4X7FXIF6VJYCZABXJB3CQJR4SYZVMKAUIA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:36.621200325 (1h26m12.660773456s) [noise=true , address=77.90.31.15:4699 , piece_id=IZDH4EVKQHBCQMBRTNW4LA55HCLZXKB3B64DJDZ4RPEFKQEW6TCA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: <nil> Sep 6 14:54:36.621233669 (1h26m12.660806789s) [seg_pos={0 0}, upload=1243]: Segment reached optimal threshold of 13 pieces. Sep 6 14:54:36.621458038 (1h26m12.661031168s) [noise=true , address=45.91.116.4:4937 , piece_id=24SIDYNY4GAPOXFKE5M3YWPRCKS4COYABJFX55TYE6THDDOYO7VQ, seg_pos={0 0}, upload=1243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:36.621566305 (1h26m12.661139435s) [noise=true , address=45.91.116.5:4297 , piece_id=GJ56CLX5HTB4LAXL3QE4234WWUJXANB4N3SUIRVMULNBWTXW5L6A, seg_pos={0 0}, upload=1243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:50752->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:50752->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50752->45.91.116.5:4297: use of closed network connection Sep 6 14:54:36.621588298 (1h26m12.661161418s) [noise=true , address=45.91.116.13:4153 , piece_id=LKUUWCFJMPPLNWSVQY4XOFVKKBA3574H7D234DE4ANZOSOMX4QCA, seg_pos={0 0}, upload=1243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:48492->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:48492->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48492->45.91.116.13:4153: use of closed network connection Sep 6 14:54:36.621640548 (1h26m12.661213678s) [seg_pos={0 0}, upload=1243]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 14:54:36.621760889 (1h26m12.661334009s) [upload=1243]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:36.621769505 (1h26m12.661342625s) [upload=1243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:37.082190940 (1h26m13.12176405s) [upload=1243]: Done uploading segments... Sep 6 14:54:37.082210087 (1h26m13.121783207s) [upload=1243]: Upload finished. err: <nil> Duration: 30.001775116s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001437028s FAIL (2025-09-06 14:54:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 1m0.001446025s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001268257s FAIL (2025-09-06 14:54:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001289338s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [X....X..X.XXXXX..X.XXXXXXXX.XX..XX.X.X.XX.XXXX.X.XX.X.XXX...] duration: 18.82303591s FAIL (2025-09-06 14:53:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:50:50.656114740 (1h22m26.69568786s) [upload=1193]: Starting upload Sep 6 14:50:50.656163453 (1h22m26.695736563s) [upload=1193]: Done starting upload Sep 6 14:50:50.656338067 (1h22m26.695911187s) [upload=1193]: Uploading segments... Sep 6 14:50:50.656371852 (1h22m26.695944962s) [upload=1193]: Got next segment. Inline: false Sep 6 14:50:50.656393383 (1h22m26.695966503s) [upload=1193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:50:50.656397551 (1h22m26.695970661s) [upload=1193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:50:50.916970362 (1h22m26.956543472s) [seg_pos={0 0}, upload=1193]: Begin upload segment... Sep 6 14:50:50.917007955 (1h22m26.956581075s) [seg_pos={0 0}, upload=1193]: Done begin upload segment. Sep 6 14:50:50.917031570 (1h22m26.95660469s) [noise=true , address=45.91.116.12:4227 , piece_id=G76FFUIOCXDORDGH6T2R3PUDE2PUDCAN5B5IJ273SPDUG5AIGNZA, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.917035437 (1h22m26.956608557s) [noise=true , address=45.91.116.5:4297 , piece_id=Y7VLW6LM4FEEXJUWY272S4KVN3NWXZGSDPNA5E6TYQSL743HMAOA, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.917070153 (1h22m26.956643273s) [noise=true , address=45.91.116.5:4298 , piece_id=GMKLHG3WAJJYAIHHYN43EQ3CRKXEORMELVFETEYHDD74YTHL6THA, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.917396788 (1h22m26.956969908s) [noise=true , address=77.90.31.10:4519 , piece_id=BZQL7PJYS2VPDX5MSQYR4HQJTXFDLZJAYLI3N5VXEUCZOURHHJCQ, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.917414122 (1h22m26.956987242s) [noise=true , address=45.91.116.10:4081 , piece_id=CKX2CI4S4LW2E4X4UZNFR6FEQ7NT4AMDC255XVBACG6QZXUJKTQQ, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.917654932 (1h22m26.957228052s) [noise=true , address=45.91.116.9:4261 , piece_id=X3KGO7OEPBVPXPQEUH2INMG7Z2L7RLKDOGTGGQWCTJQQYUOOOLUA, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.917679990 (1h22m26.9572531s) [noise=true , address=45.91.116.6:4975 , piece_id=Z7NFIX5R7IUMCASUEVAPG35IWQQZU3VSPP4NL6JPXJSUE7WZKLDQ, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.917934838 (1h22m26.957507958s) [noise=true , address=45.91.116.11:4118 , piece_id=FQPAS744ESPLJUU2JQ3N4K7KDRDXJQYGVRTHT2SDJ2EAFSVAKBDQ, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.917959556 (1h22m26.957532676s) [noise=true , address=77.90.31.15:4699 , piece_id=D5GVL7IENEU7KIAMIQTI4Z5U5FKH5JWLXIQHXJYFBQYZWXBMUWWA, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.918189535 (1h22m26.957762655s) [noise=true , address=45.91.116.12:4225 , piece_id=6H6NUQIUJOOCUT2P3FCQHFZIE3OI2PWA6PUEILZRD5P2J7PUAMLA, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.918233680 (1h22m26.9578068s) [noise=true , address=45.91.116.11:4117 , piece_id=W523LQDZ6PJ6RGC3ZSVOZGITHJ6PJW3TWOGKQANC6CVRFPP53EAQ, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.918470332 (1h22m26.958043452s) [noise=true , address=45.91.116.13:4155 , piece_id=C65QXU72NULXEAODF7VVJDEBT2BHRNIAZMQ2PRUR3UUMJHAV3BPA, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.918474360 (1h22m26.95804748s) [noise=true , address=45.91.116.5:4299 , piece_id=SFTZHMVVIU4BTAVY5JBAWSABK72RB6S3WKU65IDS6EMMKLHZXPGA, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.918680765 (1h22m26.958253885s) [noise=true , address=45.91.116.6:4973 , piece_id=WRFVMEVLWO75FA7Q4ZKVXP6LLMT2BWKOWWVHLDANQIO7DTCJGNYA, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.918859347 (1h22m26.958432467s) [noise=true , address=77.90.31.19:4841 , piece_id=CEJU4EDDOWINOIGYALBSIBYIIXUG5WLGDEQGR3TGSSQVDP5BASQQ, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:50.919126879 (1h22m26.958700029s) [noise=true , address=45.91.116.7:4009 , piece_id=APUGQ27OESHIIRSVRJIORMGPUEC2466CRFAQP4XMU3WPA5QYPFHA, seg_pos={0 0}, upload=1193]: Uploading piece... Sep 6 14:50:52.779069097 (1h22m28.818642207s) [upload=1193]: Next returned nil segment Sep 6 14:50:52.779116087 (1h22m28.818689207s) [upload=1193]: Waiting for error group managing segments... Sep 6 14:50:53.071616302 (1h22m29.111189422s) [noise=true , address=45.91.116.7:4009 , piece_id=APUGQ27OESHIIRSVRJIORMGPUEC2466CRFAQP4XMU3WPA5QYPFHA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.104372220 (1h22m29.14394534s) [noise=true , address=45.91.116.11:4118 , piece_id=FQPAS744ESPLJUU2JQ3N4K7KDRDXJQYGVRTHT2SDJ2EAFSVAKBDQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.111231005 (1h22m29.150804125s) [noise=true , address=45.91.116.9:4261 , piece_id=X3KGO7OEPBVPXPQEUH2INMG7Z2L7RLKDOGTGGQWCTJQQYUOOOLUA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.113863332 (1h22m29.153436452s) [noise=true , address=77.90.31.15:4699 , piece_id=D5GVL7IENEU7KIAMIQTI4Z5U5FKH5JWLXIQHXJYFBQYZWXBMUWWA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.145611062 (1h22m29.185184172s) [noise=true , address=45.91.116.11:4117 , piece_id=W523LQDZ6PJ6RGC3ZSVOZGITHJ6PJW3TWOGKQANC6CVRFPP53EAQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.177275823 (1h22m29.216848943s) [noise=true , address=77.90.31.19:4841 , piece_id=CEJU4EDDOWINOIGYALBSIBYIIXUG5WLGDEQGR3TGSSQVDP5BASQQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.191014221 (1h22m29.230587341s) [noise=true , address=77.90.31.10:4519 , piece_id=BZQL7PJYS2VPDX5MSQYR4HQJTXFDLZJAYLI3N5VXEUCZOURHHJCQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.217722709 (1h22m29.257295819s) [noise=true , address=45.91.116.5:4298 , piece_id=GMKLHG3WAJJYAIHHYN43EQ3CRKXEORMELVFETEYHDD74YTHL6THA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.316965101 (1h22m29.356538221s) [noise=true , address=45.91.116.10:4081 , piece_id=CKX2CI4S4LW2E4X4UZNFR6FEQ7NT4AMDC255XVBACG6QZXUJKTQQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.318382173 (1h22m29.357955293s) [noise=true , address=45.91.116.6:4973 , piece_id=WRFVMEVLWO75FA7Q4ZKVXP6LLMT2BWKOWWVHLDANQIO7DTCJGNYA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.362406144 (1h22m29.401979264s) [noise=true , address=45.91.116.12:4227 , piece_id=G76FFUIOCXDORDGH6T2R3PUDE2PUDCAN5B5IJ273SPDUG5AIGNZA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.435021605 (1h22m29.474594715s) [noise=true , address=45.91.116.6:4975 , piece_id=Z7NFIX5R7IUMCASUEVAPG35IWQQZU3VSPP4NL6JPXJSUE7WZKLDQ, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.524005989 (1h22m29.563579109s) [noise=true , address=45.91.116.5:4297 , piece_id=Y7VLW6LM4FEEXJUWY272S4KVN3NWXZGSDPNA5E6TYQSL743HMAOA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: <nil> Sep 6 14:50:53.524032830 (1h22m29.56360595s) [seg_pos={0 0}, upload=1193]: Segment reached optimal threshold of 13 pieces. Sep 6 14:50:53.524201573 (1h22m29.563774693s) [noise=true , address=45.91.116.12:4225 , piece_id=6H6NUQIUJOOCUT2P3FCQHFZIE3OI2PWA6PUEILZRD5P2J7PUAMLA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Sep 6 14:50:53.524300401 (1h22m29.563873521s) [noise=true , address=45.91.116.13:4155 , piece_id=C65QXU72NULXEAODF7VVJDEBT2BHRNIAZMQ2PRUR3UUMJHAV3BPA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Sep 6 14:50:53.524420432 (1h22m29.563993552s) [noise=true , address=45.91.116.5:4299 , piece_id=SFTZHMVVIU4BTAVY5JBAWSABK72RB6S3WKU65IDS6EMMKLHZXPGA, seg_pos={0 0}, upload=1193]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: context canceled; context canceled Sep 6 14:50:53.524431923 (1h22m29.564005043s) [seg_pos={0 0}, upload=1193]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 14:50:53.524515613 (1h22m29.564088733s) [upload=1193]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:50:53.524519962 (1h22m29.564093083s) [upload=1193]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:50:53.895318914 (1h22m29.934892034s) [upload=1193]: Done uploading segments... Sep 6 14:50:53.895337139 (1h22m29.934910259s) [upload=1193]: Upload finished. err: <nil> Duration: 18.823061929s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)