Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 7.66692512s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:54:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.666952413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X.X..............X..............................] duration: 6.864984592s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:53:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.865010851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.904014781s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:53:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.904048845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 7.18300008s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:53:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.183026711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.921250681s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:53:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.921275698s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X............X.X..............X.............................] duration: 6.723779728s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:53:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.723791772s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X............X.X..............X............................] duration: 6.844299768s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:52:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.84432215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.931402109s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:52:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.931438188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.873524133s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:52:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.873549071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.912266104s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:52:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.91229614s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X............X.X..............X...........................] duration: 6.880629152s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:51:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.880652977s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.907628135s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:51:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.907652372s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X............X.X..............X..........................] duration: 14.158984301s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:51:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.159009838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.375533766s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:51:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.375560547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.935606692s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:50:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.93563164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.556579681s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:50:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.556618315s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X............X.X..............X.........................] duration: 5.708853596s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:49:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.708873264s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.682130625s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:49:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.682154401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.459056097s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:49:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.459077027s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X............X.X..............X........................] duration: 5.862412996s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:49:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.862438865s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.613143318s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:49:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.613158145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X............X.X..............X.......................] duration: 5.819081986s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:48:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.819114037s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 8.252033656s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:48:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.252061989s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.561120125s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:48:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.561150503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X............X.X..............X......................] duration: 5.916551029s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:47:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.916573323s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 8.818499824s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:47:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.818514674s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X............X.X..............X.....................] duration: 5.760287896s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:47:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.760307544s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.769386757s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:47:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.769476668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.979980784s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:47:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.980009479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.395721994s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:47:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.395777048s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X............X.X..............X....................] duration: 5.933892269s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:46:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.933913047s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.601791777s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:46:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.601815482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.422147986s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:46:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.422173212s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X............X.X..............X...................] duration: 5.835871105s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:45:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.835893577s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 7.909697141s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:45:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.909715516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.692175449s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:45:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.69220192s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X............X.X..............X..................] duration: 6.945933034s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:45:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.94595774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.797377802s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:44:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.797399713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.709901747s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:44:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.709928077s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X............X.X..............X.................] duration: 6.822243969s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:44:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.822269558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.539847656s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:44:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.539894455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.951972335s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:43:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.952041748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.252794441s trace: https://jaeger.dev.storj.io/trace/61f95f9c55543be9 FAIL (2026-05-13 18:43:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 13 18:42:39.654115046 (311h54m50.174938791s) [upload=298824]: Starting upload May 13 18:42:39.654237830 (311h54m50.175061595s) [upload=298824]: Done starting upload May 13 18:42:39.654294318 (311h54m50.175118103s) [upload=298824]: Uploading segments... May 13 18:42:39.654647201 (311h54m50.175470986s) [upload=298824]: Got next segment. Inline: false May 13 18:42:39.654688680 (311h54m50.175512425s) [upload=298824]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 18:42:39.654695634 (311h54m50.175519379s) [upload=298824]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 18:42:39.943710770 (311h54m50.464534515s) [seg_pos={0 0}, upload=298824]: Begin upload segment... May 13 18:42:39.943793989 (311h54m50.464617765s) [seg_pos={0 0}, upload=298824]: Done begin upload segment. May 13 18:42:39.943823095 (311h54m50.46464685s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.943840539 (311h54m50.464664284s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944027826 (311h54m50.464851571s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944491761 (311h54m50.465315506s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.944901293 (311h54m50.465725038s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945324029 (311h54m50.466147774s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.945722740 (311h54m50.466546485s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946124527 (311h54m50.466948282s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.946494653 (311h54m50.467318398s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.947965089 (311h54m50.468788834s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.948256265 (311h54m50.46908001s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.950216525 (311h54m50.47104027s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.951870151 (311h54m50.472693896s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.953453131 (311h54m50.474276876s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.955147294 (311h54m50.475971039s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:39.956601288 (311h54m50.477425033s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Uploading piece... May 13 18:42:41.281556869 (311h54m51.802380614s) [upload=298824]: Next returned nil segment May 13 18:42:41.281569513 (311h54m51.802393258s) [upload=298824]: Waiting for error group managing segments... May 13 18:42:41.372457758 (311h54m51.893281503s) [noise=true , address=77.90.31.14:4691 , piece_id=JFI56VQMCJZRDD2BZ3GOSKFLNVIVHEV6U3WK7MZ6LDU77ZGPOJNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:42:41.657118546 (311h54m52.177942291s) [noise=true , address=193.149.164.71:4294 , piece_id=JNC7X62V4WU7N64M5SMBYCFFJRUSEJUV2F5EET5MEQN3CL7REKWQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.669588933 (311h54m52.190412668s) [noise=true , address=167.17.76.155:4920 , piece_id=WXCJUEM5FJLPWGXL5Y3P5W4ZFIOUI3WVN2FVY5MVQCNITDVDCRHA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.673347525 (311h54m52.19417128s) [noise=true , address=167.17.72.71:4793 , piece_id=SAEMFXJYRFHIJDBZFYAR7AXLDPHAXWTSFSIVKATGZ5SLFKQNOYVQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.809908244 (311h54m52.33073199s) [noise=true , address=145.79.227.6:4277 , piece_id=I6VBYHUGW4ENFKM6PWFNDNGVDZCZTUPWSJLTGAW4OXB3GMEUS3EA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.855212685 (311h54m52.376036441s) [noise=true , address=145.79.227.47:4282 , piece_id=UZ4CAH5KR2XUABI6KFT4QIE3HOXPP2GRRA5TXQIWST42L54244CQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.859506299 (311h54m52.380330034s) [noise=true , address=145.79.227.50:4380 , piece_id=BOO6A7PMYBGEE4PT6QUXUDBEEIE2P6G3BAHZXDKQIW5BEROQE6QA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:41.942638354 (311h54m52.463462099s) [noise=true , address=167.17.72.70:4778 , piece_id=7754ZHKMHT7GREAJKTPJ6VUOGTNU243QFPMJNIPYMEJK4QREYD4A, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.270329925 (311h54m52.79115367s) [noise=true , address=193.149.164.68:4056 , piece_id=5XNXS6DFS7EOWWKYJ5BDVFDUJPGCYQWT4L3HNH5JBMXJDH67WLXQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.305229522 (311h54m52.826053277s) [noise=true , address=193.149.164.69:4086 , piece_id=U63LKEGO2ZO6YRTQ3PKLF63VIJMMV7B6XXJH35FFQM3P6KGF7GQA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.470504164 (311h54m52.991327909s) [noise=true , address=45.91.116.14:4220 , piece_id=EDKJQP6N2GL2XKRCKNZV5TOKQK24OCKYYCLVSSBH6YHBLHWAIPNQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.529729148 (311h54m53.050552894s) [noise=true , address=45.91.116.9:4279 , piece_id=URBFWN55CEWJ6NHVUB3SABQLY7LM7IQTQUIWW2RFJVDOHNUIJRFA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.530517203 (311h54m53.051340948s) [noise=true , address=45.91.116.10:4110 , piece_id=BC5TDQDVKMQFESE5KYRT2BSS62JKEZFX33Y2IXB5S76DGNXF352Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:42:42.536522942 (311h54m53.057346687s) [noise=true , address=45.91.116.4:4942 , piece_id=STFL2OELEKJASIEZFAJ7HVRTCKKVTKY5LK5S5CYSRXZUZLQEOQDQ, seg_pos={0 0}, upload=298824]: Done uploading piece. err: <nil> May 13 18:43:38.986230390 (311h55m49.507054135s) [noise=true , address=77.90.31.15:4722 , piece_id=FHQJJBOHMFGMJ3BKJJJBOPLVATIA2AIC64XPOE25WG6IP6NN55PA, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 13 18:43:38.986322155 (311h55m49.50714591s) [noise=true , address=77.90.31.17:4791 , piece_id=2AJ2EQNEN2EAEOOO7ZWMF6BZNR5LMPYT7OGK6XQJL6NKF7NQF26Q, seg_pos={0 0}, upload=298824]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: context deadline exceeded; context deadline exceeded May 13 18:43:38.986356991 (311h55m49.507180736s) [seg_pos={0 0}, upload=298824]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 13 18:43:39.237544444 (311h55m49.758368189s) [upload=298824]: Done uploading segments... May 13 18:43:39.237578539 (311h55m49.758402284s) [upload=298824]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.25281443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X............X.X..............X................] duration: 6.923195856s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:43:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.923217768s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 4.171747884s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:43:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.171773142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............X............X.X..............X...............] duration: 6.981571953s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:42:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.981588243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 4.061016414s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:42:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.061041201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............X............X.X..............X..............] duration: 7.004195076s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:42:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.004216796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................X............X.X..............X.............] duration: 6.655665975s trace: https://jaeger.dev.storj.io/trace/340a1c5797d3638f FAIL (2026-05-13 18:41:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.655694039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.871186858s trace: https://jaeger.dev.storj.io/trace/579f3342c2bac4fe FAIL (2026-05-13 18:41:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.871214942s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)