Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
11 out of 60 is failed (threshold=3) [...X......XXX..X..X...X......X..........X.......X........X..] duration:
9.674645522s trace:
https://jaeger.dev.storj.io/trace/51269ee7190b9270
FAIL (2026-01-07 15:32:25 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.674693195s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001712612s trace:
https://jaeger.dev.storj.io/trace/404541ecf2993b82
FAIL (2026-01-07 15:32:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001744904s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
11 out of 60 is failed (threshold=3) [....X......XXX..X..X...X......X..........X.......X........X.] duration:
29.738229294s trace:
https://jaeger.dev.storj.io/trace/51269ee7190b9270
FAIL (2026-01-07 15:32:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.738276114s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.....X..X..X.XX.............X...............X.......X.......] duration:
6.266308452s trace:
https://jaeger.dev.storj.io/trace/16e9b304bd031fc3
FAIL (2026-01-07 15:32:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 15:25:47.659571758 (791h56m47.530654793s) [upload=692694]: Starting upload
Jan 7 15:25:47.659626732 (791h56m47.530709767s) [upload=692694]: Done starting upload
Jan 7 15:25:47.659741722 (791h56m47.530824767s) [upload=692694]: Uploading segments...
Jan 7 15:25:47.659844187 (791h56m47.530927222s) [upload=692694]: Got next segment. Inline: false
Jan 7 15:25:47.659899832 (791h56m47.530982867s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:25:47.659904771 (791h56m47.530987817s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:25:47.910689304 (791h56m47.781772339s) [seg_pos={0 0}, upload=692694]: Begin upload segment...
Jan 7 15:25:47.910758476 (791h56m47.781841521s) [seg_pos={0 0}, upload=692694]: Done begin upload segment.
Jan 7 15:25:47.910860731 (791h56m47.781943776s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.910884004 (791h56m47.781967039s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911417170 (791h56m47.782500215s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911713554 (791h56m47.782796599s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912226551 (791h56m47.783309596s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912552872 (791h56m47.783635907s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912960198 (791h56m47.784043233s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913440233 (791h56m47.784523288s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913517319 (791h56m47.784600364s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914242260 (791h56m47.785325295s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914497115 (791h56m47.78558015s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916417571 (791h56m47.787500616s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916496612 (791h56m47.787579657s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.918579778 (791h56m47.789662824s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919242380 (791h56m47.790325415s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919775515 (791h56m47.79085855s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:49.525362024 (791h56m49.396445069s) [upload=692694]: Next returned nil segment
Jan 7 15:25:49.525390858 (791h56m49.396473903s) [upload=692694]: Waiting for error group managing segments...
Jan 7 15:25:49.905314680 (791h56m49.776397725s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.908207539 (791h56m49.779290584s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.910530302 (791h56m49.781613347s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.912020098 (791h56m49.783103133s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.914760727 (791h56m49.785843772s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920037285 (791h56m49.79112032s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920087881 (791h56m49.791170916s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.924513478 (791h56m49.795596523s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.926219025 (791h56m49.79730206s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936334947 (791h56m49.807417992s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936763173 (791h56m49.807846208s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.308285948 (791h56m50.179368983s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352554152 (791h56m50.223637187s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352578848 (791h56m50.223661893s) [seg_pos={0 0}, upload=692694]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:25:50.352871957 (791h56m50.223955002s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.352881314 (791h56m50.223964359s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353036671 (791h56m50.224119726s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353049725 (791h56m50.22413277s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353170695 (791h56m50.22425375s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353181486 (791h56m50.224264531s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353202816 (791h56m50.224285861s) [seg_pos={0 0}, upload=692694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:25:50.353287959 (791h56m50.224371004s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:25:50.353295032 (791h56m50.224378077s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:25:50.658454673 (791h56m50.529537708s) [upload=692694]: Done uploading segments...
Jan 7 15:25:50.658477727 (791h56m50.529560762s) [upload=692694]: Upload finished. err: <nil>
Duration: 6.266339111s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXX] duration:
30.001344463s trace:
https://jaeger.dev.storj.io/trace/1e7a37ffed625cc1
FAIL (2026-01-07 15:32:07 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001411221s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
25 out of 60 is failed (threshold=3) [.X.XXXX....X....XXX..X..XXX..X...XXX.X...XX.....XX....X...XX] duration:
1m0.001563277s trace:
https://jaeger.dev.storj.io/trace/5a1f40390901c158
FAIL (2026-01-07 15:31:40 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001573677s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
17 out of 60 is failed (threshold=3) [X..X....X.X...XX..X.X..XX........X.....X..X.XX......X.X.....] duration:
38.369243397s trace:
https://jaeger.dev.storj.io/trace/6e7d706b02a6be61
FAIL (2026-01-07 15:31:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, Piece ID: X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA) context deadline exceeded; piecestore: (Node ID: 15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, Piece ID: CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ) context deadline exceeded; piecestore: (Node ID: 1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, Piece ID: ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ) context deadline exceeded
======LOGS======
:27.911963548 (791h57m27.783046584s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912566095 (791h57m27.783649131s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912978461 (791h57m27.784061526s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913303841 (791h57m27.784386876s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913592460 (791h57m27.784675505s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914036466 (791h57m27.785119531s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914078005 (791h57m27.78516104s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914358780 (791h57m27.785441815s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914368177 (791h57m27.785451222s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914638923 (791h57m27.785721968s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914644894 (791h57m27.785727929s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914996413 (791h57m27.786079448s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915107785 (791h57m27.78619083s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915355368 (791h57m27.786438413s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915537454 (791h57m27.786620489s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915665938 (791h57m27.786748973s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915932897 (791h57m27.787015932s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:29.315467031 (791h57m29.186550066s) [upload=692703]: Next returned nil segment
Jan 7 15:26:29.315481168 (791h57m29.186564203s) [upload=692703]: Waiting for error group managing segments...
Jan 7 15:26:29.779694700 (791h57m29.650777735s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.780057642 (791h57m29.651140687s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.789117924 (791h57m29.660200959s) [noise=true , address=145.79.227.40:4083 , piece_id=H5DB4CLXHFR4VE3J6W5FXQA2ECILA5OJKDVKO4LOULWLQDYO6H6A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.817859061 (791h57m29.688942096s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.837433043 (791h57m29.708516088s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.863992937 (791h57m29.735075972s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.060048646 (791h57m29.931131691s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.074487109 (791h57m29.945570154s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.111590750 (791h57m29.982673795s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.604364766 (791h57m30.475447801s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.637007453 (791h57m30.508090488s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.690921793 (791h57m30.562004838s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.718158436 (791h57m30.589241481s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734155368 (791h57m30.605238413s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734196385 (791h57m30.605279441s) [seg_pos={0 0}, upload=692703]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:26:30.734506386 (791h57m30.605589451s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734516926 (791h57m30.605599971s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734626644 (791h57m30.605709689s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734636483 (791h57m30.605719538s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734759858 (791h57m30.605842903s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734766661 (791h57m30.605849706s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734808030 (791h57m30.605891075s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734814984 (791h57m30.605898019s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734832056 (791h57m30.605915091s) [seg_pos={0 0}, upload=692703]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:26:30.734931665 (791h57m30.606014711s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:26:30.734973355 (791h57m30.6060564s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:26:31.030682109 (791h57m30.901765154s) [upload=692703]: Done uploading segments...
Jan 7 15:26:31.030705674 (791h57m30.901788709s) [upload=692703]: Upload finished. err: <nil>
Duration: 38.369271229s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00074041s trace:
https://jaeger.dev.storj.io/trace/5d9f461659804d3f
FAIL (2026-01-07 15:31:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000805424s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
11 out of 60 is failed (threshold=3) [.....X......XXX..X..X...X......X..........X.......X........X] duration:
30.000882154s trace:
https://jaeger.dev.storj.io/trace/51269ee7190b9270
FAIL (2026-01-07 15:31:23 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000905139s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000653317s trace:
https://jaeger.dev.storj.io/trace/4b53e0d1d41cc3ac
FAIL (2026-01-07 15:31:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00067634s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXX] duration:
30.001689541s trace:
https://jaeger.dev.storj.io/trace/6bbf8a00e514e7af
FAIL (2026-01-07 15:31:11 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001712344s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [......X..X..X.XX.............X...............X.......X......] duration:
6.304712829s trace:
https://jaeger.dev.storj.io/trace/16e9b304bd031fc3
FAIL (2026-01-07 15:31:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 15:25:47.659571758 (791h56m47.530654793s) [upload=692694]: Starting upload
Jan 7 15:25:47.659626732 (791h56m47.530709767s) [upload=692694]: Done starting upload
Jan 7 15:25:47.659741722 (791h56m47.530824767s) [upload=692694]: Uploading segments...
Jan 7 15:25:47.659844187 (791h56m47.530927222s) [upload=692694]: Got next segment. Inline: false
Jan 7 15:25:47.659899832 (791h56m47.530982867s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:25:47.659904771 (791h56m47.530987817s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:25:47.910689304 (791h56m47.781772339s) [seg_pos={0 0}, upload=692694]: Begin upload segment...
Jan 7 15:25:47.910758476 (791h56m47.781841521s) [seg_pos={0 0}, upload=692694]: Done begin upload segment.
Jan 7 15:25:47.910860731 (791h56m47.781943776s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.910884004 (791h56m47.781967039s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911417170 (791h56m47.782500215s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911713554 (791h56m47.782796599s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912226551 (791h56m47.783309596s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912552872 (791h56m47.783635907s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912960198 (791h56m47.784043233s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913440233 (791h56m47.784523288s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913517319 (791h56m47.784600364s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914242260 (791h56m47.785325295s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914497115 (791h56m47.78558015s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916417571 (791h56m47.787500616s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916496612 (791h56m47.787579657s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.918579778 (791h56m47.789662824s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919242380 (791h56m47.790325415s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919775515 (791h56m47.79085855s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:49.525362024 (791h56m49.396445069s) [upload=692694]: Next returned nil segment
Jan 7 15:25:49.525390858 (791h56m49.396473903s) [upload=692694]: Waiting for error group managing segments...
Jan 7 15:25:49.905314680 (791h56m49.776397725s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.908207539 (791h56m49.779290584s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.910530302 (791h56m49.781613347s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.912020098 (791h56m49.783103133s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.914760727 (791h56m49.785843772s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920037285 (791h56m49.79112032s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920087881 (791h56m49.791170916s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.924513478 (791h56m49.795596523s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.926219025 (791h56m49.79730206s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936334947 (791h56m49.807417992s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936763173 (791h56m49.807846208s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.308285948 (791h56m50.179368983s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352554152 (791h56m50.223637187s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352578848 (791h56m50.223661893s) [seg_pos={0 0}, upload=692694]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:25:50.352871957 (791h56m50.223955002s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.352881314 (791h56m50.223964359s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353036671 (791h56m50.224119726s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353049725 (791h56m50.22413277s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353170695 (791h56m50.22425375s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353181486 (791h56m50.224264531s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353202816 (791h56m50.224285861s) [seg_pos={0 0}, upload=692694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:25:50.353287959 (791h56m50.224371004s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:25:50.353295032 (791h56m50.224378077s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:25:50.658454673 (791h56m50.529537708s) [upload=692694]: Done uploading segments...
Jan 7 15:25:50.658477727 (791h56m50.529560762s) [upload=692694]: Upload finished. err: <nil>
Duration: 6.304754409s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
25 out of 60 is failed (threshold=3) [X.X.XXXX....X....XXX..X..XXX..X...XXX.X...XX.....XX....X...X] duration:
1m0.000749205s trace:
https://jaeger.dev.storj.io/trace/60d33954f6090e39
FAIL (2026-01-07 15:30:40 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000761939s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [......X......XXX..X..X...X......X..........X.......X........] duration:
20.900390954s trace:
https://jaeger.dev.storj.io/trace/20f0183cbd9611b2
FAIL (2026-01-07 15:30:18 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.900416402s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [XX..X....X.X...XX..X.X..XX........X.....X..X.XX......X.X....] duration:
26.876145487s trace:
https://jaeger.dev.storj.io/trace/6e7d706b02a6be61
FAIL (2026-01-07 15:30:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, Piece ID: X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA) context deadline exceeded; piecestore: (Node ID: 15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, Piece ID: CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ) context deadline exceeded; piecestore: (Node ID: 1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, Piece ID: ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ) context deadline exceeded
======LOGS======
:27.911963548 (791h57m27.783046584s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912566095 (791h57m27.783649131s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912978461 (791h57m27.784061526s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913303841 (791h57m27.784386876s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913592460 (791h57m27.784675505s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914036466 (791h57m27.785119531s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914078005 (791h57m27.78516104s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914358780 (791h57m27.785441815s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914368177 (791h57m27.785451222s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914638923 (791h57m27.785721968s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914644894 (791h57m27.785727929s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914996413 (791h57m27.786079448s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915107785 (791h57m27.78619083s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915355368 (791h57m27.786438413s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915537454 (791h57m27.786620489s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915665938 (791h57m27.786748973s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915932897 (791h57m27.787015932s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:29.315467031 (791h57m29.186550066s) [upload=692703]: Next returned nil segment
Jan 7 15:26:29.315481168 (791h57m29.186564203s) [upload=692703]: Waiting for error group managing segments...
Jan 7 15:26:29.779694700 (791h57m29.650777735s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.780057642 (791h57m29.651140687s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.789117924 (791h57m29.660200959s) [noise=true , address=145.79.227.40:4083 , piece_id=H5DB4CLXHFR4VE3J6W5FXQA2ECILA5OJKDVKO4LOULWLQDYO6H6A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.817859061 (791h57m29.688942096s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.837433043 (791h57m29.708516088s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.863992937 (791h57m29.735075972s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.060048646 (791h57m29.931131691s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.074487109 (791h57m29.945570154s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.111590750 (791h57m29.982673795s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.604364766 (791h57m30.475447801s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.637007453 (791h57m30.508090488s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.690921793 (791h57m30.562004838s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.718158436 (791h57m30.589241481s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734155368 (791h57m30.605238413s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734196385 (791h57m30.605279441s) [seg_pos={0 0}, upload=692703]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:26:30.734506386 (791h57m30.605589451s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734516926 (791h57m30.605599971s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734626644 (791h57m30.605709689s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734636483 (791h57m30.605719538s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734759858 (791h57m30.605842903s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734766661 (791h57m30.605849706s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734808030 (791h57m30.605891075s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734814984 (791h57m30.605898019s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734832056 (791h57m30.605915091s) [seg_pos={0 0}, upload=692703]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:26:30.734931665 (791h57m30.606014711s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:26:30.734973355 (791h57m30.6060564s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:26:31.030682109 (791h57m30.901765154s) [upload=692703]: Done uploading segments...
Jan 7 15:26:31.030705674 (791h57m30.901788709s) [upload=692703]: Upload finished. err: <nil>
Duration: 26.876166818s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXX] duration:
30.001480543s trace:
https://jaeger.dev.storj.io/trace/657134ddf18301d8
FAIL (2026-01-07 15:30:06 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001502163s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001510705s trace:
https://jaeger.dev.storj.io/trace/5c7b22359f0ff87d
FAIL (2026-01-07 15:30:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001529921s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.......X..X..X.XX.............X...............X.......X.....] duration:
5.787671941s trace:
https://jaeger.dev.storj.io/trace/16e9b304bd031fc3
FAIL (2026-01-07 15:29:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 15:25:47.659571758 (791h56m47.530654793s) [upload=692694]: Starting upload
Jan 7 15:25:47.659626732 (791h56m47.530709767s) [upload=692694]: Done starting upload
Jan 7 15:25:47.659741722 (791h56m47.530824767s) [upload=692694]: Uploading segments...
Jan 7 15:25:47.659844187 (791h56m47.530927222s) [upload=692694]: Got next segment. Inline: false
Jan 7 15:25:47.659899832 (791h56m47.530982867s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:25:47.659904771 (791h56m47.530987817s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:25:47.910689304 (791h56m47.781772339s) [seg_pos={0 0}, upload=692694]: Begin upload segment...
Jan 7 15:25:47.910758476 (791h56m47.781841521s) [seg_pos={0 0}, upload=692694]: Done begin upload segment.
Jan 7 15:25:47.910860731 (791h56m47.781943776s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.910884004 (791h56m47.781967039s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911417170 (791h56m47.782500215s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911713554 (791h56m47.782796599s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912226551 (791h56m47.783309596s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912552872 (791h56m47.783635907s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912960198 (791h56m47.784043233s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913440233 (791h56m47.784523288s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913517319 (791h56m47.784600364s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914242260 (791h56m47.785325295s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914497115 (791h56m47.78558015s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916417571 (791h56m47.787500616s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916496612 (791h56m47.787579657s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.918579778 (791h56m47.789662824s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919242380 (791h56m47.790325415s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919775515 (791h56m47.79085855s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:49.525362024 (791h56m49.396445069s) [upload=692694]: Next returned nil segment
Jan 7 15:25:49.525390858 (791h56m49.396473903s) [upload=692694]: Waiting for error group managing segments...
Jan 7 15:25:49.905314680 (791h56m49.776397725s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.908207539 (791h56m49.779290584s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.910530302 (791h56m49.781613347s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.912020098 (791h56m49.783103133s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.914760727 (791h56m49.785843772s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920037285 (791h56m49.79112032s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920087881 (791h56m49.791170916s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.924513478 (791h56m49.795596523s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.926219025 (791h56m49.79730206s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936334947 (791h56m49.807417992s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936763173 (791h56m49.807846208s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.308285948 (791h56m50.179368983s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352554152 (791h56m50.223637187s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352578848 (791h56m50.223661893s) [seg_pos={0 0}, upload=692694]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:25:50.352871957 (791h56m50.223955002s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.352881314 (791h56m50.223964359s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353036671 (791h56m50.224119726s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353049725 (791h56m50.22413277s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353170695 (791h56m50.22425375s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353181486 (791h56m50.224264531s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353202816 (791h56m50.224285861s) [seg_pos={0 0}, upload=692694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:25:50.353287959 (791h56m50.224371004s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:25:50.353295032 (791h56m50.224378077s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:25:50.658454673 (791h56m50.529537708s) [upload=692694]: Done uploading segments...
Jan 7 15:25:50.658477727 (791h56m50.529560762s) [upload=692694]: Upload finished. err: <nil>
Duration: 5.78769763s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [.......X......XXX..X..X...X......X..........X.......X.......] duration:
9.523684404s trace:
https://jaeger.dev.storj.io/trace/20f0183cbd9611b2
FAIL (2026-01-07 15:29:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.523807738s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
25 out of 60 is failed (threshold=3) [XX.X.XXXX....X....XXX..X..XXX..X...XXX.X...XX.....XX....X...] duration:
39.828255093s trace:
https://jaeger.dev.storj.io/trace/6b205cc28468a16c
FAIL (2026-01-07 15:29:40 UTC)
orbiter warning: context deadline exceeded
Duration: 39.828272977s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [.XX..X....X.X...XX..X.X..XX........X.....X..X.XX......X.X...] duration:
15.343277837s trace:
https://jaeger.dev.storj.io/trace/6e7d706b02a6be61
FAIL (2026-01-07 15:29:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, Piece ID: X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA) context deadline exceeded; piecestore: (Node ID: 15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, Piece ID: CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ) context deadline exceeded; piecestore: (Node ID: 1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, Piece ID: ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ) context deadline exceeded
======LOGS======
:27.911963548 (791h57m27.783046584s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912566095 (791h57m27.783649131s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912978461 (791h57m27.784061526s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913303841 (791h57m27.784386876s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913592460 (791h57m27.784675505s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914036466 (791h57m27.785119531s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914078005 (791h57m27.78516104s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914358780 (791h57m27.785441815s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914368177 (791h57m27.785451222s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914638923 (791h57m27.785721968s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914644894 (791h57m27.785727929s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914996413 (791h57m27.786079448s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915107785 (791h57m27.78619083s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915355368 (791h57m27.786438413s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915537454 (791h57m27.786620489s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915665938 (791h57m27.786748973s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915932897 (791h57m27.787015932s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:29.315467031 (791h57m29.186550066s) [upload=692703]: Next returned nil segment
Jan 7 15:26:29.315481168 (791h57m29.186564203s) [upload=692703]: Waiting for error group managing segments...
Jan 7 15:26:29.779694700 (791h57m29.650777735s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.780057642 (791h57m29.651140687s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.789117924 (791h57m29.660200959s) [noise=true , address=145.79.227.40:4083 , piece_id=H5DB4CLXHFR4VE3J6W5FXQA2ECILA5OJKDVKO4LOULWLQDYO6H6A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.817859061 (791h57m29.688942096s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.837433043 (791h57m29.708516088s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.863992937 (791h57m29.735075972s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.060048646 (791h57m29.931131691s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.074487109 (791h57m29.945570154s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.111590750 (791h57m29.982673795s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.604364766 (791h57m30.475447801s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.637007453 (791h57m30.508090488s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.690921793 (791h57m30.562004838s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.718158436 (791h57m30.589241481s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734155368 (791h57m30.605238413s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734196385 (791h57m30.605279441s) [seg_pos={0 0}, upload=692703]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:26:30.734506386 (791h57m30.605589451s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734516926 (791h57m30.605599971s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734626644 (791h57m30.605709689s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734636483 (791h57m30.605719538s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734759858 (791h57m30.605842903s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734766661 (791h57m30.605849706s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734808030 (791h57m30.605891075s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734814984 (791h57m30.605898019s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734832056 (791h57m30.605915091s) [seg_pos={0 0}, upload=692703]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:26:30.734931665 (791h57m30.606014711s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:26:30.734973355 (791h57m30.6060564s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:26:31.030682109 (791h57m30.901765154s) [upload=692703]: Done uploading segments...
Jan 7 15:26:31.030705674 (791h57m30.901788709s) [upload=692703]: Upload finished. err: <nil>
Duration: 15.343294087s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXX] duration:
30.000748877s trace:
https://jaeger.dev.storj.io/trace/3394c655ff2503a5
FAIL (2026-01-07 15:29:20 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000762444s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [..XX..X....X.X...XX..X.X..XX........X.....X..X.XX......X.X..] duration:
57.080631622s trace:
https://jaeger.dev.storj.io/trace/6e7d706b02a6be61
FAIL (2026-01-07 15:29:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, Piece ID: X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA) context deadline exceeded; piecestore: (Node ID: 15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, Piece ID: CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ) context deadline exceeded; piecestore: (Node ID: 1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, Piece ID: ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ) context deadline exceeded
======LOGS======
:27.911963548 (791h57m27.783046584s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912566095 (791h57m27.783649131s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912978461 (791h57m27.784061526s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913303841 (791h57m27.784386876s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913592460 (791h57m27.784675505s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914036466 (791h57m27.785119531s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914078005 (791h57m27.78516104s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914358780 (791h57m27.785441815s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914368177 (791h57m27.785451222s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914638923 (791h57m27.785721968s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914644894 (791h57m27.785727929s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914996413 (791h57m27.786079448s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915107785 (791h57m27.78619083s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915355368 (791h57m27.786438413s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915537454 (791h57m27.786620489s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915665938 (791h57m27.786748973s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915932897 (791h57m27.787015932s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:29.315467031 (791h57m29.186550066s) [upload=692703]: Next returned nil segment
Jan 7 15:26:29.315481168 (791h57m29.186564203s) [upload=692703]: Waiting for error group managing segments...
Jan 7 15:26:29.779694700 (791h57m29.650777735s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.780057642 (791h57m29.651140687s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.789117924 (791h57m29.660200959s) [noise=true , address=145.79.227.40:4083 , piece_id=H5DB4CLXHFR4VE3J6W5FXQA2ECILA5OJKDVKO4LOULWLQDYO6H6A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.817859061 (791h57m29.688942096s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.837433043 (791h57m29.708516088s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.863992937 (791h57m29.735075972s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.060048646 (791h57m29.931131691s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.074487109 (791h57m29.945570154s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.111590750 (791h57m29.982673795s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.604364766 (791h57m30.475447801s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.637007453 (791h57m30.508090488s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.690921793 (791h57m30.562004838s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.718158436 (791h57m30.589241481s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734155368 (791h57m30.605238413s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734196385 (791h57m30.605279441s) [seg_pos={0 0}, upload=692703]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:26:30.734506386 (791h57m30.605589451s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734516926 (791h57m30.605599971s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734626644 (791h57m30.605709689s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734636483 (791h57m30.605719538s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734759858 (791h57m30.605842903s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734766661 (791h57m30.605849706s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734808030 (791h57m30.605891075s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734814984 (791h57m30.605898019s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734832056 (791h57m30.605915091s) [seg_pos={0 0}, upload=692703]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:26:30.734931665 (791h57m30.606014711s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:26:30.734973355 (791h57m30.6060564s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:26:31.030682109 (791h57m30.901765154s) [upload=692703]: Done uploading segments...
Jan 7 15:26:31.030705674 (791h57m30.901788709s) [upload=692703]: Upload finished. err: <nil>
Duration: 57.080648816s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.007364034s trace:
https://jaeger.dev.storj.io/trace/32452a77b09f0be8
FAIL (2026-01-07 15:29:01 UTC)
orbiter warning: context deadline exceeded
Duration: 5.007391518s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [........X..X..X.XX.............X...............X.......X....] duration:
6.017321436s trace:
https://jaeger.dev.storj.io/trace/16e9b304bd031fc3
FAIL (2026-01-07 15:28:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 15:25:47.659571758 (791h56m47.530654793s) [upload=692694]: Starting upload
Jan 7 15:25:47.659626732 (791h56m47.530709767s) [upload=692694]: Done starting upload
Jan 7 15:25:47.659741722 (791h56m47.530824767s) [upload=692694]: Uploading segments...
Jan 7 15:25:47.659844187 (791h56m47.530927222s) [upload=692694]: Got next segment. Inline: false
Jan 7 15:25:47.659899832 (791h56m47.530982867s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:25:47.659904771 (791h56m47.530987817s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:25:47.910689304 (791h56m47.781772339s) [seg_pos={0 0}, upload=692694]: Begin upload segment...
Jan 7 15:25:47.910758476 (791h56m47.781841521s) [seg_pos={0 0}, upload=692694]: Done begin upload segment.
Jan 7 15:25:47.910860731 (791h56m47.781943776s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.910884004 (791h56m47.781967039s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911417170 (791h56m47.782500215s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911713554 (791h56m47.782796599s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912226551 (791h56m47.783309596s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912552872 (791h56m47.783635907s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912960198 (791h56m47.784043233s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913440233 (791h56m47.784523288s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913517319 (791h56m47.784600364s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914242260 (791h56m47.785325295s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914497115 (791h56m47.78558015s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916417571 (791h56m47.787500616s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916496612 (791h56m47.787579657s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.918579778 (791h56m47.789662824s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919242380 (791h56m47.790325415s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919775515 (791h56m47.79085855s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:49.525362024 (791h56m49.396445069s) [upload=692694]: Next returned nil segment
Jan 7 15:25:49.525390858 (791h56m49.396473903s) [upload=692694]: Waiting for error group managing segments...
Jan 7 15:25:49.905314680 (791h56m49.776397725s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.908207539 (791h56m49.779290584s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.910530302 (791h56m49.781613347s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.912020098 (791h56m49.783103133s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.914760727 (791h56m49.785843772s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920037285 (791h56m49.79112032s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920087881 (791h56m49.791170916s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.924513478 (791h56m49.795596523s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.926219025 (791h56m49.79730206s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936334947 (791h56m49.807417992s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936763173 (791h56m49.807846208s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.308285948 (791h56m50.179368983s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352554152 (791h56m50.223637187s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352578848 (791h56m50.223661893s) [seg_pos={0 0}, upload=692694]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:25:50.352871957 (791h56m50.223955002s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.352881314 (791h56m50.223964359s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353036671 (791h56m50.224119726s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353049725 (791h56m50.22413277s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353170695 (791h56m50.22425375s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353181486 (791h56m50.224264531s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353202816 (791h56m50.224285861s) [seg_pos={0 0}, upload=692694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:25:50.353287959 (791h56m50.224371004s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:25:50.353295032 (791h56m50.224378077s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:25:50.658454673 (791h56m50.529537708s) [upload=692694]: Done uploading segments...
Jan 7 15:25:50.658477727 (791h56m50.529560762s) [upload=692694]: Upload finished. err: <nil>
Duration: 6.017346443s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXX] duration:
30.000720887s trace:
https://jaeger.dev.storj.io/trace/7fa555caad62b36f
FAIL (2026-01-07 15:28:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000742559s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [........X......XXX..X..X...X......X..........X.......X......] duration:
17.773602634s trace:
https://jaeger.dev.storj.io/trace/20f0183cbd9611b2
FAIL (2026-01-07 15:28:48 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.773629776s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001562538s trace:
https://jaeger.dev.storj.io/trace/2f8b19690e81ffd6
FAIL (2026-01-07 15:28:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001577888s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.........X..X..X.XX.............X...............X.......X...] duration:
6.327095433s trace:
https://jaeger.dev.storj.io/trace/16e9b304bd031fc3
FAIL (2026-01-07 15:28:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 15:25:47.659571758 (791h56m47.530654793s) [upload=692694]: Starting upload
Jan 7 15:25:47.659626732 (791h56m47.530709767s) [upload=692694]: Done starting upload
Jan 7 15:25:47.659741722 (791h56m47.530824767s) [upload=692694]: Uploading segments...
Jan 7 15:25:47.659844187 (791h56m47.530927222s) [upload=692694]: Got next segment. Inline: false
Jan 7 15:25:47.659899832 (791h56m47.530982867s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:25:47.659904771 (791h56m47.530987817s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:25:47.910689304 (791h56m47.781772339s) [seg_pos={0 0}, upload=692694]: Begin upload segment...
Jan 7 15:25:47.910758476 (791h56m47.781841521s) [seg_pos={0 0}, upload=692694]: Done begin upload segment.
Jan 7 15:25:47.910860731 (791h56m47.781943776s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.910884004 (791h56m47.781967039s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911417170 (791h56m47.782500215s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911713554 (791h56m47.782796599s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912226551 (791h56m47.783309596s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912552872 (791h56m47.783635907s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912960198 (791h56m47.784043233s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913440233 (791h56m47.784523288s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913517319 (791h56m47.784600364s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914242260 (791h56m47.785325295s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914497115 (791h56m47.78558015s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916417571 (791h56m47.787500616s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916496612 (791h56m47.787579657s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.918579778 (791h56m47.789662824s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919242380 (791h56m47.790325415s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919775515 (791h56m47.79085855s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:49.525362024 (791h56m49.396445069s) [upload=692694]: Next returned nil segment
Jan 7 15:25:49.525390858 (791h56m49.396473903s) [upload=692694]: Waiting for error group managing segments...
Jan 7 15:25:49.905314680 (791h56m49.776397725s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.908207539 (791h56m49.779290584s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.910530302 (791h56m49.781613347s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.912020098 (791h56m49.783103133s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.914760727 (791h56m49.785843772s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920037285 (791h56m49.79112032s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920087881 (791h56m49.791170916s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.924513478 (791h56m49.795596523s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.926219025 (791h56m49.79730206s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936334947 (791h56m49.807417992s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936763173 (791h56m49.807846208s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.308285948 (791h56m50.179368983s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352554152 (791h56m50.223637187s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352578848 (791h56m50.223661893s) [seg_pos={0 0}, upload=692694]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:25:50.352871957 (791h56m50.223955002s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.352881314 (791h56m50.223964359s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353036671 (791h56m50.224119726s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353049725 (791h56m50.22413277s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353170695 (791h56m50.22425375s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353181486 (791h56m50.224264531s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353202816 (791h56m50.224285861s) [seg_pos={0 0}, upload=692694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:25:50.353287959 (791h56m50.224371004s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:25:50.353295032 (791h56m50.224378077s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:25:50.658454673 (791h56m50.529537708s) [upload=692694]: Done uploading segments...
Jan 7 15:25:50.658477727 (791h56m50.529560762s) [upload=692694]: Upload finished. err: <nil>
Duration: 6.327276677s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
25 out of 60 is failed (threshold=3) [.XX.X.XXXX....X....XXX..X..XXX..X...XXX.X...XX.....XX....X..] duration:
21.129051883s trace:
https://jaeger.dev.storj.io/trace/6b205cc28468a16c
FAIL (2026-01-07 15:28:27 UTC)
orbiter warning: context deadline exceeded
Duration: 21.129080828s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [...XX..X....X.X...XX..X.X..XX........X.....X..X.XX......X.X.] duration:
50.882161274s trace:
https://jaeger.dev.storj.io/trace/6e7d706b02a6be61
FAIL (2026-01-07 15:28:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, Piece ID: X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA) context deadline exceeded; piecestore: (Node ID: 15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, Piece ID: CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ) context deadline exceeded; piecestore: (Node ID: 1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, Piece ID: ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ) context deadline exceeded
======LOGS======
:27.911963548 (791h57m27.783046584s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912566095 (791h57m27.783649131s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912978461 (791h57m27.784061526s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913303841 (791h57m27.784386876s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913592460 (791h57m27.784675505s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914036466 (791h57m27.785119531s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914078005 (791h57m27.78516104s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914358780 (791h57m27.785441815s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914368177 (791h57m27.785451222s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914638923 (791h57m27.785721968s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914644894 (791h57m27.785727929s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914996413 (791h57m27.786079448s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915107785 (791h57m27.78619083s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915355368 (791h57m27.786438413s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915537454 (791h57m27.786620489s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915665938 (791h57m27.786748973s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915932897 (791h57m27.787015932s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:29.315467031 (791h57m29.186550066s) [upload=692703]: Next returned nil segment
Jan 7 15:26:29.315481168 (791h57m29.186564203s) [upload=692703]: Waiting for error group managing segments...
Jan 7 15:26:29.779694700 (791h57m29.650777735s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.780057642 (791h57m29.651140687s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.789117924 (791h57m29.660200959s) [noise=true , address=145.79.227.40:4083 , piece_id=H5DB4CLXHFR4VE3J6W5FXQA2ECILA5OJKDVKO4LOULWLQDYO6H6A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.817859061 (791h57m29.688942096s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.837433043 (791h57m29.708516088s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.863992937 (791h57m29.735075972s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.060048646 (791h57m29.931131691s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.074487109 (791h57m29.945570154s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.111590750 (791h57m29.982673795s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.604364766 (791h57m30.475447801s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.637007453 (791h57m30.508090488s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.690921793 (791h57m30.562004838s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.718158436 (791h57m30.589241481s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734155368 (791h57m30.605238413s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734196385 (791h57m30.605279441s) [seg_pos={0 0}, upload=692703]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:26:30.734506386 (791h57m30.605589451s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734516926 (791h57m30.605599971s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734626644 (791h57m30.605709689s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734636483 (791h57m30.605719538s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734759858 (791h57m30.605842903s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734766661 (791h57m30.605849706s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734808030 (791h57m30.605891075s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734814984 (791h57m30.605898019s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734832056 (791h57m30.605915091s) [seg_pos={0 0}, upload=692703]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:26:30.734931665 (791h57m30.606014711s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:26:30.734973355 (791h57m30.6060564s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:26:31.030682109 (791h57m30.901765154s) [upload=692703]: Done uploading segments...
Jan 7 15:26:31.030705674 (791h57m30.901788709s) [upload=692703]: Upload finished. err: <nil>
Duration: 50.882178296s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [.........X......XXX..X..X...X......X..........X.......X.....] duration:
10.681439535s trace:
https://jaeger.dev.storj.io/trace/20f0183cbd9611b2
FAIL (2026-01-07 15:28:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.681488008s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [..........X......XXX..X..X...X......X..........X.......X....] duration:
15.582903159s trace:
https://jaeger.dev.storj.io/trace/20f0183cbd9611b2
FAIL (2026-01-07 15:27:54 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.582928066s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001811832s trace:
https://jaeger.dev.storj.io/trace/5162b5903c25bbcd
FAIL (2026-01-07 15:27:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00183729s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
25 out of 60 is failed (threshold=3) [..XX.X.XXXX....X....XXX..X..XXX..X...XXX.X...XX.....XX....X.] duration:
40.760806476s trace:
https://jaeger.dev.storj.io/trace/6b205cc28468a16c
FAIL (2026-01-07 15:27:49 UTC)
orbiter warning: context deadline exceeded
Duration: 40.760828709s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXX] duration:
30.001239663s trace:
https://jaeger.dev.storj.io/trace/753d21f52ecf2516
FAIL (2026-01-07 15:27:43 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001262567s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..........X..X..X.XX.............X...............X.......X..] duration:
7.160992964s trace:
https://jaeger.dev.storj.io/trace/16e9b304bd031fc3
FAIL (2026-01-07 15:27:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 15:25:47.659571758 (791h56m47.530654793s) [upload=692694]: Starting upload
Jan 7 15:25:47.659626732 (791h56m47.530709767s) [upload=692694]: Done starting upload
Jan 7 15:25:47.659741722 (791h56m47.530824767s) [upload=692694]: Uploading segments...
Jan 7 15:25:47.659844187 (791h56m47.530927222s) [upload=692694]: Got next segment. Inline: false
Jan 7 15:25:47.659899832 (791h56m47.530982867s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:25:47.659904771 (791h56m47.530987817s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:25:47.910689304 (791h56m47.781772339s) [seg_pos={0 0}, upload=692694]: Begin upload segment...
Jan 7 15:25:47.910758476 (791h56m47.781841521s) [seg_pos={0 0}, upload=692694]: Done begin upload segment.
Jan 7 15:25:47.910860731 (791h56m47.781943776s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.910884004 (791h56m47.781967039s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911417170 (791h56m47.782500215s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911713554 (791h56m47.782796599s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912226551 (791h56m47.783309596s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912552872 (791h56m47.783635907s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912960198 (791h56m47.784043233s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913440233 (791h56m47.784523288s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913517319 (791h56m47.784600364s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914242260 (791h56m47.785325295s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914497115 (791h56m47.78558015s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916417571 (791h56m47.787500616s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916496612 (791h56m47.787579657s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.918579778 (791h56m47.789662824s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919242380 (791h56m47.790325415s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919775515 (791h56m47.79085855s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:49.525362024 (791h56m49.396445069s) [upload=692694]: Next returned nil segment
Jan 7 15:25:49.525390858 (791h56m49.396473903s) [upload=692694]: Waiting for error group managing segments...
Jan 7 15:25:49.905314680 (791h56m49.776397725s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.908207539 (791h56m49.779290584s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.910530302 (791h56m49.781613347s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.912020098 (791h56m49.783103133s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.914760727 (791h56m49.785843772s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920037285 (791h56m49.79112032s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920087881 (791h56m49.791170916s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.924513478 (791h56m49.795596523s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.926219025 (791h56m49.79730206s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936334947 (791h56m49.807417992s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936763173 (791h56m49.807846208s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.308285948 (791h56m50.179368983s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352554152 (791h56m50.223637187s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352578848 (791h56m50.223661893s) [seg_pos={0 0}, upload=692694]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:25:50.352871957 (791h56m50.223955002s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.352881314 (791h56m50.223964359s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353036671 (791h56m50.224119726s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353049725 (791h56m50.22413277s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353170695 (791h56m50.22425375s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353181486 (791h56m50.224264531s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353202816 (791h56m50.224285861s) [seg_pos={0 0}, upload=692694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:25:50.353287959 (791h56m50.224371004s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:25:50.353295032 (791h56m50.224378077s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:25:50.658454673 (791h56m50.529537708s) [upload=692694]: Done uploading segments...
Jan 7 15:25:50.658477727 (791h56m50.529560762s) [upload=692694]: Upload finished. err: <nil>
Duration: 7.161013914s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [....XX..X....X.X...XX..X.X..XX........X.....X..X.XX......X.X] duration:
1m0.002801643s trace:
https://jaeger.dev.storj.io/trace/6e7d706b02a6be61
FAIL (2026-01-07 15:27:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, Piece ID: X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA) context deadline exceeded; piecestore: (Node ID: 15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, Piece ID: CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ) context deadline exceeded; piecestore: (Node ID: 1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, Piece ID: ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ) context deadline exceeded
======LOGS======
:27.911963548 (791h57m27.783046584s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912566095 (791h57m27.783649131s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.912978461 (791h57m27.784061526s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913303841 (791h57m27.784386876s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.913592460 (791h57m27.784675505s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914036466 (791h57m27.785119531s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914078005 (791h57m27.78516104s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914358780 (791h57m27.785441815s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914368177 (791h57m27.785451222s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914638923 (791h57m27.785721968s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914644894 (791h57m27.785727929s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.914996413 (791h57m27.786079448s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915107785 (791h57m27.78619083s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915355368 (791h57m27.786438413s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915537454 (791h57m27.786620489s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915665938 (791h57m27.786748973s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:27.915932897 (791h57m27.787015932s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Uploading piece...
Jan 7 15:26:29.315467031 (791h57m29.186550066s) [upload=692703]: Next returned nil segment
Jan 7 15:26:29.315481168 (791h57m29.186564203s) [upload=692703]: Waiting for error group managing segments...
Jan 7 15:26:29.779694700 (791h57m29.650777735s) [noise=true , address=64.49.13.169:4705 , piece_id=QMN2TXZHJQUHS2EE6TKDDSXUUZAGVNU3LKFEOTQIUTCSHTIF3BKQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.780057642 (791h57m29.651140687s) [noise=true , address=145.79.227.6:4285 , piece_id=HBQTGJIOFHBP3HXK4KXBD7RBK3PJUYGBQSJ2RIGF76UCX7KPCDHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.789117924 (791h57m29.660200959s) [noise=true , address=145.79.227.40:4083 , piece_id=H5DB4CLXHFR4VE3J6W5FXQA2ECILA5OJKDVKO4LOULWLQDYO6H6A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.817859061 (791h57m29.688942096s) [noise=true , address=145.79.227.12:4794 , piece_id=DA2P6CRXTHYY6CIMJRDROJ6AYX4I4EGZAY5CAVL2ZGOPY3M733ZA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.837433043 (791h57m29.708516088s) [noise=true , address=145.79.227.44:4203 , piece_id=2K3H2DXLYUY3QNUY4BZA72GQYOFCBD7AFZZV2KZVVWE67IVZF3QA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:29.863992937 (791h57m29.735075972s) [noise=true , address=145.79.227.30:4764 , piece_id=2VNRPE3CKIHLCH4DMUBRUXOJ6TCKAP46U243O3TJHQQQOHPOP6FA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.060048646 (791h57m29.931131691s) [noise=true , address=77.90.31.17:4784 , piece_id=CTIFZHML4FZXM6PPFKFC5WQVVGFAUMUA2IVT3GQZ7NFNQIP4RXSQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.074487109 (791h57m29.945570154s) [noise=true , address=77.90.31.18:4825 , piece_id=JC5D4B6ZTTEKGS3YIIARFXEZBMJMPIWEWF6LJBBTJQGRNXNTH4JQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.111590750 (791h57m29.982673795s) [noise=true , address=77.90.31.13:4638 , piece_id=X7LZVU25AEGYRTL6B4GJDAC62IGODSPBBTA4SB2HDCRNSSQPUARA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.604364766 (791h57m30.475447801s) [noise=true , address=193.149.164.143:4695 , piece_id=3Y53KEFFVX6LLKU4P6OUYUCWNAX3N5TCS4XERNKOUBTMDJOE2L7Q, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.637007453 (791h57m30.508090488s) [noise=true , address=45.91.116.12:4246 , piece_id=ZFDIJXOB6EKOMPGOLPFIQ7ZNGX3KSND2DDJP4TX6WVK5YQPMQBOQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.690921793 (791h57m30.562004838s) [noise=true , address=145.79.227.49:4346 , piece_id=YR3XP2JGYVOZIMNX3B5ZGBHNWPKK5LXA3ETO6NE3QC6YEQJMX2KA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.718158436 (791h57m30.589241481s) [noise=true , address=145.79.227.33:4887 , piece_id=LWPMHYL6PFIE4562D4BBVUJUQJMRKIIY6AR2MJNAKCO7VF7T2A5A, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734155368 (791h57m30.605238413s) [noise=true , address=64.49.13.155:4916 , piece_id=KB7ZXO6TM6LHDSOADCQLSUHHMXPMKQNHWM7ATJF6C4CX436F3EMA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: <nil>
Jan 7 15:26:30.734196385 (791h57m30.605279441s) [seg_pos={0 0}, upload=692703]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:26:30.734506386 (791h57m30.605589451s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734516926 (791h57m30.605599971s) [noise=true , address=145.79.227.36:4528 , piece_id=3K4CTTAAMWGVDSO7Q2XDE2632SRWGUPWWIXMI42QIGCMS4LFURHA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734626644 (791h57m30.605709689s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734636483 (791h57m30.605719538s) [noise=true , address=145.79.227.43:4190 , piece_id=PQZKUYI6OYU4RRTFKD5O3DJ2F22EJYNY2UGNFEFRBMWREV7UVYAQ, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734759858 (791h57m30.605842903s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734766661 (791h57m30.605849706s) [noise=true , address=45.91.116.4:4963 , piece_id=BLVRNAG6R5O4A6GA6MOZJI4WWEDK57UOJOBD43OU2S3IGDB45VPA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734808030 (791h57m30.605891075s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4): protocol: expected piece hash; context canceled; EOF
Jan 7 15:26:30.734814984 (791h57m30.605898019s) [noise=true , address=145.79.227.29:4967 , piece_id=JNZVYF65SWU3LMJVHOV4XFPHXMMH3MYTFRCIOMQTONY4PBVDEBCA, seg_pos={0 0}, upload=692703]: Long tail cancellation due to optimal threshold
Jan 7 15:26:30.734832056 (791h57m30.605915091s) [seg_pos={0 0}, upload=692703]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:26:30.734931665 (791h57m30.606014711s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:26:30.734973355 (791h57m30.6060564s) [upload=692703]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:26:31.030682109 (791h57m30.901765154s) [upload=692703]: Done uploading segments...
Jan 7 15:26:31.030705674 (791h57m30.901788709s) [upload=692703]: Upload finished. err: <nil>
Duration: 1m0.002811723s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00044119s trace:
https://jaeger.dev.storj.io/trace/60b63e3f481aadc3
FAIL (2026-01-07 15:27:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000454365s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XX] duration:
30.001587752s trace:
https://jaeger.dev.storj.io/trace/545870620cb70f8d
FAIL (2026-01-07 15:27:08 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001610465s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
26 out of 60 is failed (threshold=3) [X..XX.X.XXXX....X....XXX..X..XXX..X...XXX.X...XX.....XX....X] duration:
1m0.001319489s trace:
https://jaeger.dev.storj.io/trace/6b205cc28468a16c
FAIL (2026-01-07 15:27:08 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001341411s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [...........X......XXX..X..X...X......X..........X.......X...] duration:
29.081057814s trace:
https://jaeger.dev.storj.io/trace/20f0183cbd9611b2
FAIL (2026-01-07 15:27:08 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.081089024s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [X..........X..X..X.XX.............X...............X.......X.] duration:
5.760668754s trace:
https://jaeger.dev.storj.io/trace/16e9b304bd031fc3
FAIL (2026-01-07 15:27:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 15:25:47.659571758 (791h56m47.530654793s) [upload=692694]: Starting upload
Jan 7 15:25:47.659626732 (791h56m47.530709767s) [upload=692694]: Done starting upload
Jan 7 15:25:47.659741722 (791h56m47.530824767s) [upload=692694]: Uploading segments...
Jan 7 15:25:47.659844187 (791h56m47.530927222s) [upload=692694]: Got next segment. Inline: false
Jan 7 15:25:47.659899832 (791h56m47.530982867s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:25:47.659904771 (791h56m47.530987817s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:25:47.910689304 (791h56m47.781772339s) [seg_pos={0 0}, upload=692694]: Begin upload segment...
Jan 7 15:25:47.910758476 (791h56m47.781841521s) [seg_pos={0 0}, upload=692694]: Done begin upload segment.
Jan 7 15:25:47.910860731 (791h56m47.781943776s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.910884004 (791h56m47.781967039s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911417170 (791h56m47.782500215s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911713554 (791h56m47.782796599s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912226551 (791h56m47.783309596s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912552872 (791h56m47.783635907s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912960198 (791h56m47.784043233s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913440233 (791h56m47.784523288s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913517319 (791h56m47.784600364s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914242260 (791h56m47.785325295s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914497115 (791h56m47.78558015s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916417571 (791h56m47.787500616s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916496612 (791h56m47.787579657s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.918579778 (791h56m47.789662824s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919242380 (791h56m47.790325415s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919775515 (791h56m47.79085855s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:49.525362024 (791h56m49.396445069s) [upload=692694]: Next returned nil segment
Jan 7 15:25:49.525390858 (791h56m49.396473903s) [upload=692694]: Waiting for error group managing segments...
Jan 7 15:25:49.905314680 (791h56m49.776397725s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.908207539 (791h56m49.779290584s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.910530302 (791h56m49.781613347s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.912020098 (791h56m49.783103133s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.914760727 (791h56m49.785843772s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920037285 (791h56m49.79112032s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920087881 (791h56m49.791170916s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.924513478 (791h56m49.795596523s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.926219025 (791h56m49.79730206s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936334947 (791h56m49.807417992s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936763173 (791h56m49.807846208s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.308285948 (791h56m50.179368983s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352554152 (791h56m50.223637187s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352578848 (791h56m50.223661893s) [seg_pos={0 0}, upload=692694]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:25:50.352871957 (791h56m50.223955002s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.352881314 (791h56m50.223964359s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353036671 (791h56m50.224119726s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353049725 (791h56m50.22413277s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353170695 (791h56m50.22425375s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353181486 (791h56m50.224264531s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353202816 (791h56m50.224285861s) [seg_pos={0 0}, upload=692694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:25:50.353287959 (791h56m50.224371004s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:25:50.353295032 (791h56m50.224378077s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:25:50.658454673 (791h56m50.529537708s) [upload=692694]: Done uploading segments...
Jan 7 15:25:50.658477727 (791h56m50.529560762s) [upload=692694]: Upload finished. err: <nil>
Duration: 5.760700445s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001406531s trace:
https://jaeger.dev.storj.io/trace/e9393abcc79157
FAIL (2026-01-07 15:26:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001433033s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
17 out of 60 is failed (threshold=3) [.....XX..X....X.X...XX..X.X..XX........X.....X..X.XX......X.] duration:
57.478055846s trace:
https://jaeger.dev.storj.io/trace/eb8f12f3928d5fe
FAIL (2026-01-07 15:26:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo, Piece ID: VHD76OCCJXTQGYSGLZ24TKYFUJZDAF73L22OXMV6GBXPH6GVTJJQ) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: SBRJS5AKJPFTLDRMESXLNJ46PEKL5MFANMFU3EJCD7UQSZJ3WLEA) context deadline exceeded; piecestore: (Node ID: 12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, Piece ID: CIQ5DMC3TL3FXOWWUZJMTR3DURFYFITZQ5AWQIWS5FE37YWDL2PQ) context deadline exceeded; piecestore: (Node ID: 1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, Piece ID: 52NXPLXY5BWYMXV7B7DDK3CCESBSGLALSBOM7WEKLEGO33K4UPOA) context deadline exceeded
======LOGS======
4993 , piece_id=JUF5PC5AB4HPK2QKYO4JIL62PYKKXO4DCAOLCNZILFGAOLTKODDA, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.418156705 (791h55m24.289239751s) [noise=true , address=77.90.31.15:4731 , piece_id=VHD76OCCJXTQGYSGLZ24TKYFUJZDAF73L22OXMV6GBXPH6GVTJJQ, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.418441237 (791h55m24.289524272s) [noise=true , address=45.91.116.11:4126 , piece_id=DENIMYOWAJ7PCI6RLREETY5PSME7G5SS52INW5LEYM5IGGTGRURQ, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.418745677 (791h55m24.289828722s) [noise=true , address=145.79.227.35:4516 , piece_id=VDZFHJJOFWL6MYMAYHSTYRVCFLEWH6C646IDMNXB3N2NXCGJ3YGA, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.419056269 (791h55m24.290139304s) [noise=true , address=145.79.227.33:4893 , piece_id=B7VY4MHRE2QIESBVQK3Q5SKGZXVVKAHKPHHBVMNLWSBBMC6BDOMQ, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.419380766 (791h55m24.290463811s) [noise=true , address=145.79.227.43:4178 , piece_id=T64WEKOOSUJT4RZOBQG4U2XJ6D46O67JBNCGH2ZKDBOXN7N7IQQQ, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.421296654 (791h55m24.292379699s) [noise=true , address=64.49.13.155:4922 , piece_id=IB5HNPNB2F62IYPBSNAD5MI2ALFRUQ3ILUUEWCEURUDBXIQJTPFQ, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.421789342 (791h55m24.292872377s) [noise=true , address=64.49.13.169:4715 , piece_id=FM6LYZKTRDWCFDKCE3DNPUV3VLSYBV7ZWKRQYA3BYRCROJGNHSHQ, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.425767978 (791h55m24.296851023s) [noise=true , address=77.90.31.16:4765 , piece_id=77EMVS3WLZRMN7BOC44D34CSPTRZE6U4MEDB5EG7M5ML656CQGCQ, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.427724203 (791h55m24.298807248s) [noise=true , address=77.90.31.11:4567 , piece_id=I7A3JJPFI46PQBNVOSHEVENRLQMRNSBIJITPGKBLXVEWLEHM3I3Q, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.429399925 (791h55m24.30048296s) [noise=true , address=77.90.31.12:4617 , piece_id=MMBCVJLML4MIUXLD6WGZM3UGREXZBFDIDELQFBWY3CM7K5D5MT4Q, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.431188259 (791h55m24.302271305s) [noise=true , address=145.79.227.30:4784 , piece_id=2YR2ERY65XHW2V62N34RITSNFH24JMN4X53QKMSL2UO62NCX4OLA, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.432436687 (791h55m24.303519722s) [noise=true , address=145.79.227.40:4072 , piece_id=OHX3NFTVLU7WN6NMRQRNZZHY4SU2PW6M5KVMMPQE7YT6KU2YILJA, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.433774965 (791h55m24.30485801s) [noise=true , address=145.79.227.4:4206 , piece_id=WB3VXNCSKRL7JOK26ECWFM3BTUVSFNJOBN5U4EGXAKUFOXVNIOOA, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:24.434596449 (791h55m24.305679494s) [noise=true , address=45.91.116.5:4314 , piece_id=SBRJS5AKJPFTLDRMESXLNJ46PEKL5MFANMFU3EJCD7UQSZJ3WLEA, seg_pos={0 0}, upload=692676]: Uploading piece...
Jan 7 15:24:25.769379302 (791h55m25.640462347s) [upload=692676]: Next returned nil segment
Jan 7 15:24:25.769392188 (791h55m25.640475223s) [upload=692676]: Waiting for error group managing segments...
Jan 7 15:24:26.230597882 (791h55m26.101680917s) [noise=true , address=64.49.13.169:4715 , piece_id=FM6LYZKTRDWCFDKCE3DNPUV3VLSYBV7ZWKRQYA3BYRCROJGNHSHQ, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:26.272146006 (791h55m26.143229051s) [noise=true , address=145.79.227.43:4178 , piece_id=T64WEKOOSUJT4RZOBQG4U2XJ6D46O67JBNCGH2ZKDBOXN7N7IQQQ, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:26.281057766 (791h55m26.152140811s) [noise=true , address=145.79.227.40:4072 , piece_id=OHX3NFTVLU7WN6NMRQRNZZHY4SU2PW6M5KVMMPQE7YT6KU2YILJA, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:26.295542387 (791h55m26.166625422s) [noise=true , address=145.79.227.35:4516 , piece_id=VDZFHJJOFWL6MYMAYHSTYRVCFLEWH6C646IDMNXB3N2NXCGJ3YGA, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:26.323541210 (791h55m26.194624255s) [noise=true , address=145.79.227.30:4784 , piece_id=2YR2ERY65XHW2V62N34RITSNFH24JMN4X53QKMSL2UO62NCX4OLA, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:26.343925785 (791h55m26.21500882s) [noise=true , address=145.79.227.34:4914 , piece_id=6TDNY4S2EZKEUARFL5QT3JUTGSF2L7MRZWVDE6Y47H3TE5A72JTA, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:26.597167496 (791h55m26.468250531s) [noise=true , address=77.90.31.11:4567 , piece_id=I7A3JJPFI46PQBNVOSHEVENRLQMRNSBIJITPGKBLXVEWLEHM3I3Q, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:26.721073382 (791h55m26.592156427s) [noise=true , address=45.91.116.5:4314 , piece_id=SBRJS5AKJPFTLDRMESXLNJ46PEKL5MFANMFU3EJCD7UQSZJ3WLEA, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:26.835136776 (791h55m26.706219811s) [noise=true , address=45.91.116.12:4252 , piece_id=52NXPLXY5BWYMXV7B7DDK3CCESBSGLALSBOM7WEKLEGO33K4UPOA, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:26.854084365 (791h55m26.72516741s) [noise=true , address=45.91.116.11:4126 , piece_id=DENIMYOWAJ7PCI6RLREETY5PSME7G5SS52INW5LEYM5IGGTGRURQ, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:26.982160230 (791h55m26.853243275s) [noise=true , address=64.49.13.155:4922 , piece_id=IB5HNPNB2F62IYPBSNAD5MI2ALFRUQ3ILUUEWCEURUDBXIQJTPFQ, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:27.281536206 (791h55m27.152619251s) [noise=true , address=145.79.227.33:4893 , piece_id=B7VY4MHRE2QIESBVQK3Q5SKGZXVVKAHKPHHBVMNLWSBBMC6BDOMQ, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:27.298321788 (791h55m27.169404833s) [noise=true , address=77.90.31.15:4731 , piece_id=VHD76OCCJXTQGYSGLZ24TKYFUJZDAF73L22OXMV6GBXPH6GVTJJQ, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:27.394379475 (791h55m27.26546252s) [noise=true , address=77.90.31.13:4652 , piece_id=CIQ5DMC3TL3FXOWWUZJMTR3DURFYFITZQ5AWQIWS5FE37YWDL2PQ, seg_pos={0 0}, upload=692676]: Done uploading piece. err: <nil>
Jan 7 15:24:27.394402179 (791h55m27.265485224s) [seg_pos={0 0}, upload=692676]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:24:27.394632127 (791h55m27.265715162s) [noise=true , address=145.79.227.4:4206 , piece_id=WB3VXNCSKRL7JOK26ECWFM3BTUVSFNJOBN5U4EGXAKUFOXVNIOOA, seg_pos={0 0}, upload=692676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ih3m2bU4PfDMX5zfXoGw3t2eyY1u6wVnpqczo2sz7aTq99R9T): protocol: write tcp 188.166.225.170:40470->145.79.227.4:4206: use of closed network connection; write tcp 188.166.225.170:40470->145.79.227.4:4206: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40470->145.79.227.4:4206: use of closed network connection
Jan 7 15:24:27.394639311 (791h55m27.265722346s) [noise=true , address=145.79.227.4:4206 , piece_id=WB3VXNCSKRL7JOK26ECWFM3BTUVSFNJOBN5U4EGXAKUFOXVNIOOA, seg_pos={0 0}, upload=692676]: Long tail cancellation due to optimal threshold
Jan 7 15:24:27.394705948 (791h55m27.265789013s) [noise=true , address=77.90.31.16:4765 , piece_id=77EMVS3WLZRMN7BOC44D34CSPTRZE6U4MEDB5EG7M5ML656CQGCQ, seg_pos={0 0}, upload=692676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CSrtv2fMmFBx9zje9zBevpVzxqyhH7fL7VnHbBFEYFRPuwCKK): protocol: expected piece hash; context canceled; EOF
Jan 7 15:24:27.394761272 (791h55m27.265844458s) [noise=true , address=77.90.31.16:4765 , piece_id=77EMVS3WLZRMN7BOC44D34CSPTRZE6U4MEDB5EG7M5ML656CQGCQ, seg_pos={0 0}, upload=692676]: Long tail cancellation due to optimal threshold
Jan 7 15:24:27.394885710 (791h55m27.265968745s) [noise=true , address=77.90.31.12:4617 , piece_id=MMBCVJLML4MIUXLD6WGZM3UGREXZBFDIDELQFBWY3CM7K5D5MT4Q, seg_pos={0 0}, upload=692676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF
Jan 7 15:24:27.394892563 (791h55m27.265975608s) [noise=true , address=77.90.31.12:4617 , piece_id=MMBCVJLML4MIUXLD6WGZM3UGREXZBFDIDELQFBWY3CM7K5D5MT4Q, seg_pos={0 0}, upload=692676]: Long tail cancellation due to optimal threshold
Jan 7 15:24:27.394983556 (791h55m27.266066591s) [noise=true , address=45.91.116.6:4993 , piece_id=JUF5PC5AB4HPK2QKYO4JIL62PYKKXO4DCAOLCNZILFGAOLTKODDA, seg_pos={0 0}, upload=692676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:51528->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:51528->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51528->45.91.116.6:4993: use of closed network connection
Jan 7 15:24:27.394990189 (791h55m27.266073224s) [noise=true , address=45.91.116.6:4993 , piece_id=JUF5PC5AB4HPK2QKYO4JIL62PYKKXO4DCAOLCNZILFGAOLTKODDA, seg_pos={0 0}, upload=692676]: Long tail cancellation due to optimal threshold
Jan 7 15:24:27.395078217 (791h55m27.266161252s) [seg_pos={0 0}, upload=692676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:24:27.395259472 (791h55m27.266342517s) [upload=692676]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:24:27.395267346 (791h55m27.266350392s) [upload=692676]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:24:27.688627787 (791h55m27.559710832s) [upload=692676]: Done uploading segments...
Jan 7 15:24:27.688646914 (791h55m27.559729949s) [upload=692676]: Upload finished. err: <nil>
Duration: 57.478085093s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00128558s trace:
https://jaeger.dev.storj.io/trace/12a00512642241f
FAIL (2026-01-07 15:26:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001299156s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.X..........X..X..X.XX.............X...............X.......X] duration:
30.004102615s trace:
https://jaeger.dev.storj.io/trace/16e9b304bd031fc3
FAIL (2026-01-07 15:26:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 7 15:25:47.659571758 (791h56m47.530654793s) [upload=692694]: Starting upload
Jan 7 15:25:47.659626732 (791h56m47.530709767s) [upload=692694]: Done starting upload
Jan 7 15:25:47.659741722 (791h56m47.530824767s) [upload=692694]: Uploading segments...
Jan 7 15:25:47.659844187 (791h56m47.530927222s) [upload=692694]: Got next segment. Inline: false
Jan 7 15:25:47.659899832 (791h56m47.530982867s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:25:47.659904771 (791h56m47.530987817s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:25:47.910689304 (791h56m47.781772339s) [seg_pos={0 0}, upload=692694]: Begin upload segment...
Jan 7 15:25:47.910758476 (791h56m47.781841521s) [seg_pos={0 0}, upload=692694]: Done begin upload segment.
Jan 7 15:25:47.910860731 (791h56m47.781943776s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.910884004 (791h56m47.781967039s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911417170 (791h56m47.782500215s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.911713554 (791h56m47.782796599s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912226551 (791h56m47.783309596s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912552872 (791h56m47.783635907s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.912960198 (791h56m47.784043233s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913440233 (791h56m47.784523288s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.913517319 (791h56m47.784600364s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914242260 (791h56m47.785325295s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.914497115 (791h56m47.78558015s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916417571 (791h56m47.787500616s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.916496612 (791h56m47.787579657s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.918579778 (791h56m47.789662824s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919242380 (791h56m47.790325415s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:47.919775515 (791h56m47.79085855s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Uploading piece...
Jan 7 15:25:49.525362024 (791h56m49.396445069s) [upload=692694]: Next returned nil segment
Jan 7 15:25:49.525390858 (791h56m49.396473903s) [upload=692694]: Waiting for error group managing segments...
Jan 7 15:25:49.905314680 (791h56m49.776397725s) [noise=true , address=145.79.227.14:4840 , piece_id=QBETP43OGZIH5CGENNAPNSANDK3PIMKZVAPNVMSLBKDG4HQKTO7A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.908207539 (791h56m49.779290584s) [noise=true , address=145.79.227.6:4259 , piece_id=B5QCLSL36AAH4VGZB3VRF33K7MIRE6JEE6NE3GIKGIRZ5JKIKLTQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.910530302 (791h56m49.781613347s) [noise=true , address=145.79.227.19:4077 , piece_id=HRG54OFSTS65SYJTGQOAJJMVKQI337QH3B3RK7LBFGYCFVR3ULKA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.912020098 (791h56m49.783103133s) [noise=true , address=145.79.227.37:4565 , piece_id=2FNQABGGBAEFEZ2HHBQJRGOFBBME7IUQZWIAZN3S2GXWCHEJQ4HA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.914760727 (791h56m49.785843772s) [noise=true , address=145.79.227.31:4799 , piece_id=HS3VX6FRH4MIEQRNH3SH3S7IF4QBB4VDNBFPZFHITSCAJJG3FQEA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920037285 (791h56m49.79112032s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7UNJGRXQI5CS5Q7QG3RVWGLQSBMLVVCLZ33M45XOIDIB7OTTKVQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.920087881 (791h56m49.791170916s) [noise=true , address=145.79.227.42:4123 , piece_id=HCBGDBP5ZU2GFKBGYSCSKFZKJEVO5USN3Y6O23BB2WU3ZDNJMP2Q, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.924513478 (791h56m49.795596523s) [noise=true , address=145.79.227.30:4762 , piece_id=TKECC3DBXVC4PCAOLTK5WMRSLVBPGBCA7PB67IGFHLMXDK4IANNQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.926219025 (791h56m49.79730206s) [noise=true , address=145.79.227.41:4085 , piece_id=CZNXS442HANTZ26CH5ZFW4G4SC4AUMQAGN33YKU4WRW57VIWR75A, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936334947 (791h56m49.807417992s) [noise=true , address=145.79.227.26:4653 , piece_id=QCSZD2LHUKH4F2H4J67K5H4VEHA2OS5DKM6JVTY3MD2PGSM2P3PA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:49.936763173 (791h56m49.807846208s) [noise=true , address=145.79.227.47:4265 , piece_id=Q34VJRKCFEIS3LHMGQR2AR7D3BOGFJNUWGT2BPWLLXQC5YMIFBBQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.308285948 (791h56m50.179368983s) [noise=true , address=77.90.31.21:4877 , piece_id=PSFFYW4LW4YF75ND2LW24WUBAAZ74IV424BMMQW5QJTHJ25GU5OQ, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352554152 (791h56m50.223637187s) [noise=true , address=45.91.116.6:4976 , piece_id=URGUFANECE24RUSUTS52WZT7GUQ7YCDURHH5RDIBJ3MM7OJBPDVA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: <nil>
Jan 7 15:25:50.352578848 (791h56m50.223661893s) [seg_pos={0 0}, upload=692694]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:25:50.352871957 (791h56m50.223955002s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.352881314 (791h56m50.223964359s) [noise=true , address=45.91.116.14:4192 , piece_id=VFHT3BCFQ2U4LGRHQYNSVYNCDZ4YR2DDZEZL2R3MZJPUPI4ILPWA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353036671 (791h56m50.224119726s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353049725 (791h56m50.22413277s) [noise=true , address=45.91.116.7:4009 , piece_id=X3ILFRQQ4CSIOHVMGKV6XBYBBBHPO5LQV5M45USGKK6OLRHQATHA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353170695 (791h56m50.22425375s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Jan 7 15:25:50.353181486 (791h56m50.224264531s) [noise=true , address=45.91.116.10:4083 , piece_id=6AQW65OVWHEZ4CLRTHF5TMMF5D6KTFVRR7ES2OJXPV6CY7BNZVJA, seg_pos={0 0}, upload=692694]: Long tail cancellation due to optimal threshold
Jan 7 15:25:50.353202816 (791h56m50.224285861s) [seg_pos={0 0}, upload=692694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:25:50.353287959 (791h56m50.224371004s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:25:50.353295032 (791h56m50.224378077s) [upload=692694]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:25:50.658454673 (791h56m50.529537708s) [upload=692694]: Done uploading segments...
Jan 7 15:25:50.658477727 (791h56m50.529560762s) [upload=692694]: Upload finished. err: <nil>
Duration: 30.004123886s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.X] duration:
30.001192006s trace:
https://jaeger.dev.storj.io/trace/7c794458f2e66641
FAIL (2026-01-07 15:26:16 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001217624s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [............X......XXX..X..X...X......X..........X.......X..] duration:
15.334264436s trace:
https://jaeger.dev.storj.io/trace/20f0183cbd9611b2
FAIL (2026-01-07 15:26:02 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.334299402s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000487019s trace:
https://jaeger.dev.storj.io/trace/37b797b740a1c1e1
FAIL (2026-01-07 15:25:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000514542s
core
uplink
us-select-1
us1-placement-20
|