Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX...XX.XX.....] duration: 3.548112213s trace: https://jaeger.dev.storj.io/trace/b95637ccae85751 FAIL (2025-12-15 08:49:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.548685676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......X......X.................................] duration: 11.379630212s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:49:11 UTC) orbiter warning: context deadline exceeded Duration: 11.379658398s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.......X......X................................] duration: 11.904497364s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:48:54 UTC) orbiter warning: context deadline exceeded Duration: 11.904524495s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..............X.................] duration: 9.078717543s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:48:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 9.078747299s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX...XX.XX....] duration: 3.425322958s trace: https://jaeger.dev.storj.io/trace/b95637ccae85751 FAIL (2025-12-15 08:48:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.425352765s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.......X......X...............................] duration: 11.970954236s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:48:11 UTC) orbiter warning: context deadline exceeded Duration: 11.971001637s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X..............X................] duration: 8.32407374s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:48:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 8.324105851s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX...XX.XX...] duration: 5.000938876s trace: https://jaeger.dev.storj.io/trace/b95637ccae85751 FAIL (2025-12-15 08:47:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000962781s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......X......X..............................] duration: 13.507848523s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:47:16 UTC) orbiter warning: context deadline exceeded Duration: 13.507879762s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X..............X...............] duration: 7.894532081s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:47:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 7.894561457s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX...XX.XX..] duration: 3.513358796s trace: https://jaeger.dev.storj.io/trace/b95637ccae85751 FAIL (2025-12-15 08:46:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513387771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X..............X..............] duration: 7.650324219s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:46:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 7.650353384s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX...XX.XX.] duration: 3.517419461s trace: https://jaeger.dev.storj.io/trace/b95637ccae85751 FAIL (2025-12-15 08:45:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517530252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......X......X.............................] duration: 5.652520331s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:45:53 UTC) orbiter warning: context deadline exceeded Duration: 5.652551161s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.......X......X............................] duration: 5.776286774s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:45:24 UTC) orbiter warning: context deadline exceeded Duration: 5.776354371s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..............X.............] duration: 8.919501408s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:45:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 8.91953381s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX...XX.XX] duration: 5.002053495s trace: https://jaeger.dev.storj.io/trace/b95637ccae85751 FAIL (2025-12-15 08:45:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002133366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X..............X............] duration: 27.471794103s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:44:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 27.471817667s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......X......X...........................] duration: 12.813195669s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:44:20 UTC) orbiter warning: context deadline exceeded Duration: 12.813224704s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX...XX.X] duration: 5.000889213s trace: https://jaeger.dev.storj.io/trace/71d7184ea2dcb175 FAIL (2025-12-15 08:44:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000916334s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX...XX.] duration: 3.605971191s trace: https://jaeger.dev.storj.io/trace/34aeac40ec4f813f FAIL (2025-12-15 08:43:32 UTC) orbiter warning: context deadline exceeded Duration: 3.606004474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X..............X...........] duration: 8.482685658s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:43:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 8.482714422s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X...X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX...XX] duration: 5.000930871s trace: https://jaeger.dev.storj.io/trace/34aeac40ec4f813f FAIL (2025-12-15 08:43:07 UTC) orbiter warning: context deadline exceeded Duration: 5.000960198s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......X......X..........................] duration: 5.866843951s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:43:05 UTC) orbiter warning: context deadline exceeded Duration: 5.866874009s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.......X......X.........................] duration: 22.102136037s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:42:41 UTC) orbiter warning: context deadline exceeded Duration: 22.102166405s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X..............X..........] duration: 9.15014795s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:42:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 9.15017936s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X...X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX...X] duration: 5.001522082s trace: https://jaeger.dev.storj.io/trace/252afb2705e482e3 FAIL (2025-12-15 08:42:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001566417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X..............X.........] duration: 8.827924982s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:42:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 8.827955329s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [..X...X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX...] duration: 3.571090771s trace: https://jaeger.dev.storj.io/trace/39079d2c2c945a5c FAIL (2025-12-15 08:42:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.571119387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.......X......X........................] duration: 17.419437281s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:41:44 UTC) orbiter warning: context deadline exceeded Duration: 17.419467668s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..............X........] duration: 9.005857363s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:41:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 9.005886148s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.......X......X.......................] duration: 16.239620784s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:41:14 UTC) orbiter warning: context deadline exceeded Duration: 16.239653475s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...X...X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX..] duration: 3.664343584s trace: https://jaeger.dev.storj.io/trace/39079d2c2c945a5c FAIL (2025-12-15 08:41:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.664359424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..............X.......] duration: 12.331714259s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:40:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 12.331741821s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [....X...X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX.] duration: 3.567019626s trace: https://jaeger.dev.storj.io/trace/39079d2c2c945a5c FAIL (2025-12-15 08:40:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.567049524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.......X......X......................] duration: 18.379678075s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:40:13 UTC) orbiter warning: context deadline exceeded Duration: 18.379695808s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......X......X.....................] duration: 19.296249895s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:39:55 UTC) orbiter warning: context deadline exceeded Duration: 19.296284201s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..............X......] duration: 10.027911646s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:39:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 10.027944218s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..XX] duration: 5.001667862s trace: https://jaeger.dev.storj.io/trace/39079d2c2c945a5c FAIL (2025-12-15 08:39:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001693832s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.......X......X....................] duration: 17.725949638s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:38:48 UTC) orbiter warning: context deadline exceeded Duration: 17.72598142s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X..............X.....] duration: 8.879201688s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:38:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 8.879232587s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......X......X...................] duration: 24.526893653s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:38:25 UTC) orbiter warning: context deadline exceeded Duration: 24.526921936s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..X] duration: 5.001303742s trace: https://jaeger.dev.storj.io/trace/3c097d5230eca9a1 FAIL (2025-12-15 08:38:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001378755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..............X....] duration: 8.145389382s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:37:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 8.145421744s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX..] duration: 3.495649441s trace: https://jaeger.dev.storj.io/trace/24745d0c26f608f6 FAIL (2025-12-15 08:37:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.495676583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X..............X...] duration: 8.793845279s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:37:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 8.79387144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X..............X..] duration: 7.998627218s trace: https://jaeger.dev.storj.io/trace/31eea2cd44408b46 FAIL (2025-12-15 08:37:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA) context deadline exceeded ======LOGS====== 6m41.620642946s) [upload=204400]: Uploading segments... Dec 15 08:35:41.749725727 (233h6m41.620808772s) [upload=204400]: Got next segment. Inline: false Dec 15 08:35:41.749769611 (233h6m41.620852656s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:35:41.749775572 (233h6m41.620858617s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:35:41.992670654 (233h6m41.86375372s) [seg_pos={0 0}, upload=204400]: Begin upload segment... Dec 15 08:35:41.992745407 (233h6m41.863828452s) [seg_pos={0 0}, upload=204400]: Done begin upload segment. Dec 15 08:35:41.992783689 (233h6m41.863866734s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992794971 (233h6m41.863878006s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.992876547 (233h6m41.863959592s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993327235 (233h6m41.86441028s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.993727446 (233h6m41.864810481s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.994173746 (233h6m41.865256791s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995523246 (233h6m41.866606291s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.995918458 (233h6m41.867001504s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:41.999068237 (233h6m41.870151272s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.002351047 (233h6m41.873434113s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.005070175 (233h6m41.87615324s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.010131293 (233h6m41.881214338s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.011655886 (233h6m41.882738931s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.012994094 (233h6m41.884077139s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.013491912 (233h6m41.884574957s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:42.016784742 (233h6m41.887867787s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Uploading piece... Dec 15 08:35:44.001012761 (233h6m43.872095816s) [upload=204400]: Next returned nil segment Dec 15 08:35:44.001097753 (233h6m43.872180808s) [upload=204400]: Waiting for error group managing segments... Dec 15 08:35:44.314284565 (233h6m44.185367611s) [noise=true , address=45.91.116.10:4082 , piece_id=7MRWQXZAD7TW3BBKPFCY3YJXWQFH4D46HQVXIDPDRJQHPFSOZ6ZA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.330509360 (233h6m44.201592415s) [noise=true , address=77.90.31.10:4519 , piece_id=WTKCYNCFRWDBSATLUZUPQ2U6Y5B5IJVJHNRHWP344K377CQIJ7BQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.615571067 (233h6m44.486654102s) [noise=true , address=145.79.227.48:4301 , piece_id=KOKQGBZ2JKESV6WL4H7P34W4TWAWR62FIPTMQAQX2NX5W77Y2JAQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.627612005 (233h6m44.4986951s) [noise=true , address=145.79.227.46:4603 , piece_id=KSQPIXL2IDKCMSB5OIIX35T5QJ76F5OGIEGHUOAY5A7TLWYPHHVA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.634966249 (233h6m44.506049314s) [noise=true , address=145.79.227.35:4481 , piece_id=LPJGIK6VYVXOQQMV6TCSM7RYXQ3DTTZTBJMQDZYOPPA3GKUCD4HQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.635622348 (233h6m44.506705393s) [noise=true , address=145.79.227.41:4086 , piece_id=WEFK5K3BJTGY5YNTFODXA3IAAKO3RNKRJC5NUZXAH26R3I5AXBUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641106782 (233h6m44.512189817s) [noise=true , address=145.79.227.22:4510 , piece_id=INMR66C6IWWESRFFOFQV3LZ6ABDQ2JCDCCUHD4HRNP3QSF5GVIHQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641357449 (233h6m44.512440484s) [noise=true , address=145.79.227.51:4409 , piece_id=MEZPJXTJSY6MWPCLK2TLX2WBKKDU62NOLPJF4MKTHST5H4VIQ4LQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.641947993 (233h6m44.513031039s) [noise=true , address=145.79.227.36:4517 , piece_id=3M25JLTDJ5SJGQ3XSCCP5IONI7QSOSXTQ3FU5BQKS3GJWZYJYMJQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.642481820 (233h6m44.513564855s) [noise=true , address=145.79.227.47:4267 , piece_id=SLAHDW2NTYOP6NTQDRKBYZ4NCH6XAQETBV2XZTJEXIJ7TYD6YZUA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.650015415 (233h6m44.521098461s) [noise=true , address=145.79.227.50:4373 , piece_id=2DCT33O7CA74RT2IGJNWRXJAVTIPRJFSDT2B2WHKGYBSEYDHCGIQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.666877795 (233h6m44.53796084s) [noise=true , address=145.79.227.26:4655 , piece_id=RTTZMAN4CFT6UDSNUQRWER55TRJGGRNIHTSRTO4PABUSFCK2BTGQ, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.676981545 (233h6m44.54806459s) [noise=true , address=145.79.227.45:4231 , piece_id=ZBU2TWLMME4GIQXC65Z57FFUFHKWDT6JXUV6TCOKC76ZGUQEZ42Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: <nil> Dec 15 08:35:44.677007775 (233h6m44.54809082s) [seg_pos={0 0}, upload=204400]: Segment reached optimal threshold of 13 pieces. Dec 15 08:35:44.677287889 (233h6m44.548370934s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 08:35:44.677299490 (233h6m44.548382535s) [noise=true , address=145.79.227.32:4836 , piece_id=ZI5QH4UHV63PBSCQFPGQ3KCXAZGGP5NKFOTP24OZMEQKETNW6A2A, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677489582 (233h6m44.548572627s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34296->145.79.227.8:4329: use of closed network connection Dec 15 08:35:44.677503679 (233h6m44.548586714s) [noise=true , address=145.79.227.8:4329 , piece_id=UBHSFH7LSKHUE2RZJREY5BWDHLH4SKKJIGK5FE6DEV2AZUAMQLFA, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677698069 (233h6m44.548781104s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53974->145.79.227.20:4438: use of closed network connection Dec 15 08:35:44.677709000 (233h6m44.548792035s) [noise=true , address=145.79.227.20:4438 , piece_id=OND5LRTVXEGA6NTIFQQIXRF26DEPHOTJCQTZ7U3SJMN4GEA3UJ4Q, seg_pos={0 0}, upload=204400]: Long tail cancellation due to optimal threshold Dec 15 08:35:44.677756540 (233h6m44.548839586s) [seg_pos={0 0}, upload=204400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:35:44.677917447 (233h6m44.549000492s) [upload=204400]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:35:44.677924711 (233h6m44.549007776s) [upload=204400]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:35:45.009262377 (233h6m44.880345432s) [upload=204400]: Done uploading segments... Dec 15 08:35:45.009286522 (233h6m44.880369567s) [upload=204400]: Upload finished. err: <nil> Duration: 7.998661305s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......X......X..................] duration: 7.150825851s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:37:00 UTC) orbiter warning: context deadline exceeded Duration: 7.150893731s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......X......X.................] duration: 22.202234081s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:36:53 UTC) orbiter warning: context deadline exceeded Duration: 22.202260931s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X.......XXXX.] duration: 3.634787362s trace: https://jaeger.dev.storj.io/trace/24745d0c26f608f6 FAIL (2025-12-15 08:36:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.634816367s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)