Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
13 out of 60 is failed (threshold=3) [....X......X..X.X.X........X.....XX......X..........XX...X.X] duration:
1m0.003012399s trace:
https://jaeger.dev.storj.io/trace/76df23b0035fe317
FAIL (2026-01-06 16:40:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
g piece...
Jan 6 16:39:13.319055742 (769h10m13.190138787s) [noise=true , address=77.90.31.12:4616 , piece_id=JCWWA7PGIW23J4OKRJYW33S37FGDTIHBHTCLJSXXA4RH4SE7SWUA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.319538962 (769h10m13.190621997s) [noise=true , address=45.91.116.7:4014 , piece_id=65DKIVFKLQVVJEHZOT2P4KYDBHFD6URVGOFPFPABEGTOORENHXBA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.320135148 (769h10m13.191218193s) [noise=true , address=77.90.31.14:4668 , piece_id=QYCFWBXHW7BYLILL76W5MDFYQJKG6VZIZOYS3BF4OPDZLMJZXKVQ, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.320651351 (769h10m13.191734396s) [noise=true , address=145.79.227.11:4736 , piece_id=CTSU3X6OZJFWJSCLJBJ3EBGALWWGZL62ZJN436LKICHA4MLIQ2ZA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.321172283 (769h10m13.192255328s) [noise=true , address=64.49.13.27:4968 , piece_id=X4RAE7BUCM2R5NDMCCLH4TRGRCCXJBGUXZRE4QSNCFWNCJXLBSVA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.323456752 (769h10m13.194539787s) [noise=true , address=145.79.227.12:4773 , piece_id=JXMIF6EZKPD7EXZ646Z4XM6PG6I5P3EPTBJUWYQX4LMGAUFJT4EA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.325413879 (769h10m13.196496924s) [noise=true , address=145.79.227.33:4884 , piece_id=E5YFUCC75VTQBLZH3T5Q6VWEMRYAVOXNFKZQCPEP3ES64YCTXWPA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.327557260 (769h10m13.198640305s) [noise=true , address=145.79.227.6:4279 , piece_id=FNXBVE7QJDR67LLYLBXOCK3OQ7E6O5BIKKXPQMYOSMKIZSNK2EDQ, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.329566295 (769h10m13.20064934s) [noise=true , address=45.91.116.9:4282 , piece_id=GC72TML4SY7YTFQW4G3CVTOMGW5HWQLCWHBFQ3TKSZSUJUR252WA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.330791288 (769h10m13.201874323s) [noise=true , address=45.91.116.11:4139 , piece_id=NRCGKCNT7VWR4UUJE7HLXWT23K3BRUV2AHMDPOAVAXQF4UZKIPUA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.332349425 (769h10m13.20343246s) [noise=true , address=145.79.227.49:4371 , piece_id=YNFO36VNDUUPFQO4OFHIUEWUZWLYM2BUD7NF6VQEVWZHNCDSM6FA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.332771599 (769h10m13.203854634s) [noise=true , address=145.79.227.23:4557 , piece_id=QHDDTZ4DZAJCIU3I5HVA54JQLOHCCN73GSJAWZGC5JL6X4WV7ETA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.333853448 (769h10m13.204936494s) [noise=true , address=145.79.227.14:4843 , piece_id=GSANAVILLP6UTITEPDCBK5VHTTK7D2SBT7FQQI77H35I3SLNRVKA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:13.336038750 (769h10m13.207121785s) [noise=true , address=45.91.116.10:4107 , piece_id=I5OP5FXQBURDQG2LWEKGVZUBX7VV3CLD6Z2ZKKDK5UCX2RGFX3BA, seg_pos={0 0}, upload=672957]: Uploading piece...
Jan 6 16:39:14.793189300 (769h10m14.664272335s) [upload=672957]: Next returned nil segment
Jan 6 16:39:14.793201272 (769h10m14.664284307s) [upload=672957]: Waiting for error group managing segments...
Jan 6 16:39:15.209058789 (769h10m15.080141834s) [noise=true , address=145.79.227.13:4835 , piece_id=MMTT3F5VEI2NUZ4T2QWJGNCPIB3O4EGLYANWMCFK6MC3YLQX3TXA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:15.251874898 (769h10m15.122957933s) [noise=true , address=145.79.227.14:4843 , piece_id=GSANAVILLP6UTITEPDCBK5VHTTK7D2SBT7FQQI77H35I3SLNRVKA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:15.299718689 (769h10m15.170801724s) [noise=true , address=145.79.227.11:4736 , piece_id=CTSU3X6OZJFWJSCLJBJ3EBGALWWGZL62ZJN436LKICHA4MLIQ2ZA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:15.351276473 (769h10m15.222359518s) [noise=true , address=145.79.227.12:4773 , piece_id=JXMIF6EZKPD7EXZ646Z4XM6PG6I5P3EPTBJUWYQX4LMGAUFJT4EA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:15.423449492 (769h10m15.294532527s) [noise=true , address=64.49.13.27:4968 , piece_id=X4RAE7BUCM2R5NDMCCLH4TRGRCCXJBGUXZRE4QSNCFWNCJXLBSVA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:15.580777038 (769h10m15.451860083s) [noise=true , address=77.90.31.14:4668 , piece_id=QYCFWBXHW7BYLILL76W5MDFYQJKG6VZIZOYS3BF4OPDZLMJZXKVQ, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:15.798468394 (769h10m15.66955143s) [noise=true , address=45.91.116.10:4107 , piece_id=I5OP5FXQBURDQG2LWEKGVZUBX7VV3CLD6Z2ZKKDK5UCX2RGFX3BA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:16.083877398 (769h10m15.954960443s) [noise=true , address=45.91.116.9:4282 , piece_id=GC72TML4SY7YTFQW4G3CVTOMGW5HWQLCWHBFQ3TKSZSUJUR252WA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:16.143050622 (769h10m16.014133687s) [noise=true , address=145.79.227.32:4865 , piece_id=Z5ZJEDUJG376CVJM4EDK6JPRJK2U5OR6RDD4YG5X4MXH4WKSZYSA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:16.194307403 (769h10m16.065390438s) [noise=true , address=145.79.227.33:4884 , piece_id=E5YFUCC75VTQBLZH3T5Q6VWEMRYAVOXNFKZQCPEP3ES64YCTXWPA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:16.244019653 (769h10m16.115102688s) [noise=true , address=145.79.227.49:4371 , piece_id=YNFO36VNDUUPFQO4OFHIUEWUZWLYM2BUD7NF6VQEVWZHNCDSM6FA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:16.587195683 (769h10m16.458278718s) [noise=true , address=45.91.116.7:4014 , piece_id=65DKIVFKLQVVJEHZOT2P4KYDBHFD6URVGOFPFPABEGTOORENHXBA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:16.752240113 (769h10m16.623323148s) [noise=true , address=145.79.227.21:4501 , piece_id=KQIHUVS3N56R5HFH2S2I6OE7PBZUXAHRGFMPK57KX7UA5I7BLCSQ, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:16.752279798 (769h10m16.623362843s) [noise=true , address=77.90.31.12:4616 , piece_id=JCWWA7PGIW23J4OKRJYW33S37FGDTIHBHTCLJSXXA4RH4SE7SWUA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: <nil>
Jan 6 16:39:16.752291660 (769h10m16.623374695s) [seg_pos={0 0}, upload=672957]: Segment reached optimal threshold of 14 pieces.
Jan 6 16:39:16.752502462 (769h10m16.623585507s) [noise=true , address=145.79.227.6:4279 , piece_id=FNXBVE7QJDR67LLYLBXOCK3OQ7E6O5BIKKXPQMYOSMKIZSNK2EDQ, seg_pos={0 0}, upload=672957]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: write tcp 188.166.225.170:35010->145.79.227.6:4279: use of closed network connection; write tcp 188.166.225.170:35010->145.79.227.6:4279: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35010->145.79.227.6:4279: use of closed network connection
Jan 6 16:39:16.752510658 (769h10m16.623593693s) [noise=true , address=145.79.227.6:4279 , piece_id=FNXBVE7QJDR67LLYLBXOCK3OQ7E6O5BIKKXPQMYOSMKIZSNK2EDQ, seg_pos={0 0}, upload=672957]: Long tail cancellation due to optimal threshold
Jan 6 16:39:16.752633512 (769h10m16.623716557s) [noise=true , address=145.79.227.20:4454 , piece_id=W7E7EUDKE6BTN5I7TSTAMHETYKCXBU7RLYALUHELLOSTOAUFVB2Q, seg_pos={0 0}, upload=672957]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J): protocol: expected piece hash; context canceled; EOF
Jan 6 16:39:16.752641587 (769h10m16.623724632s) [noise=true , address=145.79.227.20:4454 , piece_id=W7E7EUDKE6BTN5I7TSTAMHETYKCXBU7RLYALUHELLOSTOAUFVB2Q, seg_pos={0 0}, upload=672957]: Long tail cancellation due to optimal threshold
Jan 6 16:39:16.752764281 (769h10m16.623847316s) [noise=true , address=145.79.227.23:4557 , piece_id=QHDDTZ4DZAJCIU3I5HVA54JQLOHCCN73GSJAWZGC5JL6X4WV7ETA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sF7TCu1pL8mrtjoPULJ2VnZQHcJSMmvou2w7YgF58ZHQcnfJT): protocol: write tcp 188.166.225.170:43038->145.79.227.23:4557: use of closed network connection; write tcp 188.166.225.170:43038->145.79.227.23:4557: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43038->145.79.227.23:4557: use of closed network connection
Jan 6 16:39:16.752774249 (769h10m16.623857295s) [noise=true , address=145.79.227.23:4557 , piece_id=QHDDTZ4DZAJCIU3I5HVA54JQLOHCCN73GSJAWZGC5JL6X4WV7ETA, seg_pos={0 0}, upload=672957]: Long tail cancellation due to optimal threshold
Jan 6 16:39:16.752797775 (769h10m16.62388081s) [noise=true , address=45.91.116.11:4139 , piece_id=NRCGKCNT7VWR4UUJE7HLXWT23K3BRUV2AHMDPOAVAXQF4UZKIPUA, seg_pos={0 0}, upload=672957]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD): protocol: read tcp 188.166.225.170:44870->45.91.116.11:4139: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 16:39:16.752805439 (769h10m16.623888484s) [noise=true , address=45.91.116.11:4139 , piece_id=NRCGKCNT7VWR4UUJE7HLXWT23K3BRUV2AHMDPOAVAXQF4UZKIPUA, seg_pos={0 0}, upload=672957]: Long tail cancellation due to optimal threshold
Jan 6 16:39:16.752825017 (769h10m16.623908052s) [seg_pos={0 0}, upload=672957]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 16:39:16.752902604 (769h10m16.623985659s) [upload=672957]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:39:16.752918193 (769h10m16.624001229s) [upload=672957]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:39:17.044336649 (769h10m16.915419684s) [upload=672957]: Done uploading segments...
Jan 6 16:39:17.044357138 (769h10m16.915440173s) [upload=672957]: Upload finished. err: <nil>
Duration: 1m0.003026065s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XX.XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.006551275s trace:
https://jaeger.dev.storj.io/trace/3cbb5f019ee4452
FAIL (2026-01-06 16:39:58 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.006572947s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X...X..X.XXXX..X.....XXX........X.......X..XX.....XX.X...X.X] duration:
30.002775996s trace:
https://jaeger.dev.storj.io/trace/1e8504951ab4e2f5
FAIL (2026-01-06 16:39:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 6 16:39:24.581971333 (769h10m24.453054378s) [upload=672961]: Starting upload
Jan 6 16:39:24.582058469 (769h10m24.453141504s) [upload=672961]: Done starting upload
Jan 6 16:39:24.582335857 (769h10m24.453418902s) [upload=672961]: Uploading segments...
Jan 6 16:39:24.582391253 (769h10m24.453474298s) [upload=672961]: Got next segment. Inline: false
Jan 6 16:39:24.582433833 (769h10m24.453516869s) [upload=672961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 16:39:24.582439504 (769h10m24.453522539s) [upload=672961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 16:39:24.826953076 (769h10m24.698036111s) [seg_pos={0 0}, upload=672961]: Begin upload segment...
Jan 6 16:39:24.827075489 (769h10m24.698158534s) [seg_pos={0 0}, upload=672961]: Done begin upload segment.
Jan 6 16:39:24.827121126 (769h10m24.698204161s) [noise=true , address=145.79.227.46:4603 , piece_id=PXGP4ANMZNILDAKU2IJWL5UOLYFHIIFQITIZARW7D24YSWCKBMCQ, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.827213011 (769h10m24.698296046s) [noise=true , address=145.79.227.3:4151 , piece_id=A4FWQ7YW6IN3H7WNNVJ4CUA2HCZ3B23CK3YWFZQLTVFUTGVJZ3UA, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.828172238 (769h10m24.699255283s) [noise=true , address=77.90.31.17:4773 , piece_id=665JKWBTXGU5I6ZQF47ZMNCX7CAK2JY3CJUYKFG5MPWEFTFIDQWA, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.828593850 (769h10m24.699676885s) [noise=true , address=145.79.227.11:4731 , piece_id=5O2X3ER64V6KUCHOE5EAHZTD5YHF6OOFSGTEKVQAXUD6WK5QM5MQ, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.828613347 (769h10m24.699696382s) [noise=true , address=145.79.227.10:4403 , piece_id=LORJ6G2Z5DUHHWLQ752ETBJZYMU4KNNX2XMXPZHOQTBN25DFYPYQ, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.828927906 (769h10m24.700010941s) [noise=true , address=77.90.31.21:4877 , piece_id=YHD5WXPFTVYRB4QM2CGTANIQZ4WA3VKL2LVC2QU3PD2UTZ7L7QHA, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.829058946 (769h10m24.700141981s) [noise=true , address=77.90.31.11:4554 , piece_id=EZN342BHAF25TYIL4RNUJQYJM527QUTJPI7NTENVIFOOYHJ4C7MQ, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.829225824 (769h10m24.700308859s) [noise=true , address=45.91.116.5:4299 , piece_id=QQTMIGI6LIPCUXBW2AJ7FXHV7LEOKG6LNRC7UQP7AUGAFQZANNCQ, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.829403021 (769h10m24.700486056s) [noise=true , address=45.91.116.14:4191 , piece_id=7VEBLYVJE2RUZM6NTKXBAAARTWQR5ELBIKKABHAF7LYKBTZ2XYIA, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.829509163 (769h10m24.700592198s) [noise=true , address=145.79.227.16:4912 , piece_id=MLS4N24CBXXYL4JE2JSTPCK5OJ3PFJNHMTFTL5N7S5IN6YSLKLFQ, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.829777394 (769h10m24.700860449s) [noise=true , address=145.79.227.52:4447 , piece_id=BLELR2WSJXJXDRZBGVDVARFA6MBMIBZNUZU3B4XWDN2253PDEEIQ, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.829789266 (769h10m24.700872301s) [noise=true , address=77.90.31.15:4697 , piece_id=ZKRFWNDQDDOCUHZVTQEKWLRE6SYSJCVHMUFLQGJ5X4W35SKB23JQ, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.830112281 (769h10m24.701195316s) [noise=true , address=145.79.227.31:4799 , piece_id=VTLXJKCMXUWZ3KT5NIDOU7FJRNDE73RPA6SP5BHLFMYYIZAJHN3A, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.830495911 (769h10m24.701578947s) [noise=true , address=45.91.116.8:4045 , piece_id=DRAX6GQLWPTNOWE6NQDTKFMOVSEMVUDT24PJXWC7RMMIMZUDFLRQ, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.830975154 (769h10m24.702058189s) [noise=true , address=145.79.227.5:4223 , piece_id=F5WPVTVRRN5CDP2XQB64VS25ME6O4LKKW53OX7BH76RJUU5VXYOQ, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:24.833739899 (769h10m24.704822934s) [noise=true , address=145.79.227.45:4229 , piece_id=KNMMKFFU2WMNZXYGPHHQC32F6SYDCBVADRZSMNQ57WKXLL6DOL5Q, seg_pos={0 0}, upload=672961]: Uploading piece...
Jan 6 16:39:26.534357086 (769h10m26.405440121s) [upload=672961]: Next returned nil segment
Jan 6 16:39:26.534376923 (769h10m26.405459958s) [upload=672961]: Waiting for error group managing segments...
Jan 6 16:39:26.817409961 (769h10m26.688492996s) [noise=true , address=145.79.227.16:4912 , piece_id=MLS4N24CBXXYL4JE2JSTPCK5OJ3PFJNHMTFTL5N7S5IN6YSLKLFQ, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:26.818968940 (769h10m26.690051975s) [noise=true , address=145.79.227.10:4403 , piece_id=LORJ6G2Z5DUHHWLQ752ETBJZYMU4KNNX2XMXPZHOQTBN25DFYPYQ, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:26.820225873 (769h10m26.691308908s) [noise=true , address=145.79.227.46:4603 , piece_id=PXGP4ANMZNILDAKU2IJWL5UOLYFHIIFQITIZARW7D24YSWCKBMCQ, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:26.821011770 (769h10m26.692094805s) [noise=true , address=145.79.227.11:4731 , piece_id=5O2X3ER64V6KUCHOE5EAHZTD5YHF6OOFSGTEKVQAXUD6WK5QM5MQ, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:26.823718774 (769h10m26.694801809s) [noise=true , address=145.79.227.3:4151 , piece_id=A4FWQ7YW6IN3H7WNNVJ4CUA2HCZ3B23CK3YWFZQLTVFUTGVJZ3UA, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:26.838256837 (769h10m26.709339873s) [noise=true , address=145.79.227.31:4799 , piece_id=VTLXJKCMXUWZ3KT5NIDOU7FJRNDE73RPA6SP5BHLFMYYIZAJHN3A, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:26.838737101 (769h10m26.709820136s) [noise=true , address=145.79.227.5:4223 , piece_id=F5WPVTVRRN5CDP2XQB64VS25ME6O4LKKW53OX7BH76RJUU5VXYOQ, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:26.903588428 (769h10m26.774671463s) [noise=true , address=145.79.227.52:4447 , piece_id=BLELR2WSJXJXDRZBGVDVARFA6MBMIBZNUZU3B4XWDN2253PDEEIQ, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:26.988367832 (769h10m26.859450867s) [noise=true , address=145.79.227.45:4229 , piece_id=KNMMKFFU2WMNZXYGPHHQC32F6SYDCBVADRZSMNQ57WKXLL6DOL5Q, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:27.266379128 (769h10m27.137462193s) [noise=true , address=45.91.116.14:4191 , piece_id=7VEBLYVJE2RUZM6NTKXBAAARTWQR5ELBIKKABHAF7LYKBTZ2XYIA, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:27.272259375 (769h10m27.14334241s) [noise=true , address=45.91.116.8:4045 , piece_id=DRAX6GQLWPTNOWE6NQDTKFMOVSEMVUDT24PJXWC7RMMIMZUDFLRQ, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:27.280133408 (769h10m27.151216443s) [noise=true , address=45.91.116.5:4299 , piece_id=QQTMIGI6LIPCUXBW2AJ7FXHV7LEOKG6LNRC7UQP7AUGAFQZANNCQ, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:27.290668669 (769h10m27.161751735s) [noise=true , address=77.90.31.21:4877 , piece_id=YHD5WXPFTVYRB4QM2CGTANIQZ4WA3VKL2LVC2QU3PD2UTZ7L7QHA, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:27.290697203 (769h10m27.161780258s) [seg_pos={0 0}, upload=672961]: Segment reached optimal threshold of 13 pieces.
Jan 6 16:39:27.290946078 (769h10m27.162029123s) [noise=true , address=77.90.31.11:4554 , piece_id=EZN342BHAF25TYIL4RNUJQYJM527QUTJPI7NTENVIFOOYHJ4C7MQ, seg_pos={0 0}, upload=672961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:48198->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:48198->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->77.90.31.11:4554: use of closed network connection
Jan 6 16:39:27.290953401 (769h10m27.162036446s) [noise=true , address=77.90.31.11:4554 , piece_id=EZN342BHAF25TYIL4RNUJQYJM527QUTJPI7NTENVIFOOYHJ4C7MQ, seg_pos={0 0}, upload=672961]: Long tail cancellation due to optimal threshold
Jan 6 16:39:27.291096564 (769h10m27.162179609s) [noise=true , address=77.90.31.15:4697 , piece_id=ZKRFWNDQDDOCUHZVTQEKWLRE6SYSJCVHMUFLQGJ5X4W35SKB23JQ, seg_pos={0 0}, upload=672961]: Done uploading piece. err: <nil>
Jan 6 16:39:27.291214718 (769h10m27.162297764s) [noise=true , address=77.90.31.17:4773 , piece_id=665JKWBTXGU5I6ZQF47ZMNCX7CAK2JY3CJUYKFG5MPWEFTFIDQWA, seg_pos={0 0}, upload=672961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:33694->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:33694->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33694->77.90.31.17:4773: use of closed network connection
Jan 6 16:39:27.291223476 (769h10m27.162306511s) [noise=true , address=77.90.31.17:4773 , piece_id=665JKWBTXGU5I6ZQF47ZMNCX7CAK2JY3CJUYKFG5MPWEFTFIDQWA, seg_pos={0 0}, upload=672961]: Long tail cancellation due to optimal threshold
Jan 6 16:39:27.291241379 (769h10m27.162324415s) [seg_pos={0 0}, upload=672961]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jan 6 16:39:27.291324679 (769h10m27.162407714s) [upload=672961]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:39:27.291329398 (769h10m27.162412433s) [upload=672961]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:39:27.603649453 (769h10m27.474732498s) [upload=672961]: Done uploading segments...
Jan 6 16:39:27.603675832 (769h10m27.474758877s) [upload=672961]: Upload finished. err: <nil>
Duration: 30.002798189s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [..XX......................X.X....X.X.X....X.X...X....XXX....] duration:
22.657164433s trace:
https://jaeger.dev.storj.io/trace/73dd0c7413e4b545
FAIL (2026-01-06 16:39:53 UTC)
orbiter warning: context deadline exceeded
Duration: 22.657647563s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...X......................X......X..........................] duration:
16.176920131s trace:
https://jaeger.dev.storj.io/trace/d5e2d8b8127d900
FAIL (2026-01-06 16:39:39 UTC)
orbiter warning: context deadline exceeded
Duration: 16.176941962s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XX.XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001274114s trace:
https://jaeger.dev.storj.io/trace/1755e2cb38c39f9
FAIL (2026-01-06 16:39:32 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001296797s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration:
11.257544504s trace:
https://jaeger.dev.storj.io/trace/4b3b841c440bfc62
FAIL (2026-01-06 16:39:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913312874 (769h5m40.784395909s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913698277 (769h5m40.784781322s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913862180 (769h5m40.784945215s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914346662 (769h5m40.785429707s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914842456 (769h5m40.785925491s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915307461 (769h5m40.786390506s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915793817 (769h5m40.786876863s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916279973 (769h5m40.787363018s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916759336 (769h5m40.787842371s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.920089677 (769h5m40.791172722s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:41.741185400 (769h5m41.612268445s) [upload=672894]: Next returned nil segment
Jan 6 16:34:41.741201020 (769h5m41.612284055s) [upload=672894]: Waiting for error group managing segments...
Jan 6 16:34:42.136024704 (769h5m42.00710774s) [noise=true , address=185.167.204.205:4036 , piece_id=4XKAIZPEJZAVAS7P2QILKY7XIDIIRIRWH7HTN5OVXUHAL2Y3H4SQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137413950 (769h5m42.008496995s) [noise=true , address=185.167.204.206:4051 , piece_id=XWKJM6ZIHISLSESJ2ZQQQMZTGMK5MHH6L4ZD5RXJ3SMAVASVVVOA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137527506 (769h5m42.008610541s) [noise=true , address=185.167.204.206:4049 , piece_id=4T665GS7XTBVEHUVT6SFIQW4F3TTBPDKZ57PKQHHHWWKC4P4JQPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.138183485 (769h5m42.00926653s) [noise=true , address=185.167.204.206:4048 , piece_id=ZGQYUYRK5PK24UF4PYT67FHNPR4GVFDCJ2OEGRFENLB2GYR2LFVQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.139902609 (769h5m42.010985644s) [noise=true , address=185.167.204.206:4056 , piece_id=KYHDW5RLJTZCURXCFIP7HN3ZYLPPHK7FTO6IIGQ5VJJUPF4ICQXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.140277843 (769h5m42.011360878s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.142394303 (769h5m42.013477348s) [noise=true , address=185.167.204.205:4021 , piece_id=U4X3NZRT7MHG2JTNRVDRYIC72UDMHUQUCY4VFW7JFVLLVACZN7OA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.179529514 (769h5m42.050612549s) [noise=true , address=185.167.204.206:4053 , piece_id=HBFCBJCOLPJLNE4INHP4JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.194387988 (769h5m42.065471023s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.197860059 (769h5m42.068943094s) [noise=true , address=185.167.204.206:4064 , piece_id=JGCXE3QUORVKLRIWEXFZUCT4XUF3DRVA35ZKQU4DQODLZBOH5I4A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.827615690 (769h5m42.698698725s) [noise=true , address=81.171.14.70:4041 , piece_id=NLGMNQJDFISSAHBX3EC6WMWJLP74FMR4MRPUIX4GCCCQRG5RRRCQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.917676689 (769h5m42.788759725s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.932213350 (769h5m42.803296385s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.956051121 (769h5m42.827134196s) [noise=true , address=81.171.14.70:4031 , piece_id=OJKJHNVWTPDITZC5CTQNB7UFPIVZDPE54ZXS4LWRM3XHYAB6443A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.005259852 (769h5m42.876342887s) [noise=true , address=185.167.204.205:4019 , piece_id=I7OP7AQS2DC3C7GZWG5RVJSLDKAZD2434WMBPRYMZFVAJYJG3JHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.052708792 (769h5m42.923791837s) [noise=true , address=81.171.14.70:4046 , piece_id=OV3PMCD6UIFNSQNTCOS3GEQC4676KFJOUH5ZX576OFOYC5EXCZPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.639749858 (769h5m43.510832903s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.719270598 (769h5m43.590353643s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.892052420 (769h5m43.763135466s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.901787498 (769h5m43.772870533s) [noise=true , address=81.171.14.70:4039 , piece_id=VFXOSS5DRL43RDJGPLSH3667HWKAF5ZNC2CF5BA7TDR4NSCG75HA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.019887208 (769h5m43.890970243s) [noise=true , address=185.167.204.206:4050 , piece_id=KSCVNGXUGFV7XRNU5MFIKLLQGZ5BNHTIXQLVHWPMITZOQUA23DGQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.510337921 (769h5m44.381420957s) [noise=true , address=185.167.204.205:4028 , piece_id=QPB4EPPEENW277S4SBIOCWWMBLIE6GKCGEM3TV3MXN6SYFMQAI7Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.692198242 (769h5m44.563281277s) [noise=true , address=89.149.201.147:4010 , piece_id=C57ORVBIZE4QUWCVG3OVG3KW2Z2GYVWUXEOYG5R53KV2WUQLR7JQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.394574949 (769h5m46.265657984s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.543864889 (769h5m46.414947934s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846238946 (769h5m46.717321982s) [noise=true , address=185.167.204.205:4006 , piece_id=W3XXEI7MG37VJHHK2CECNKIIHB7WSQABGVB5W5ZGAWSTFQ4NQMEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846258273 (769h5m46.717341308s) [seg_pos={0 0}, upload=672894]: Segment reached optimal threshold of 26 pieces.
Jan 6 16:34:46.846448986 (769h5m46.717532021s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846454998 (769h5m46.717538033s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846579175 (769h5m46.71766221s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846595475 (769h5m46.71767851s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846620453 (769h5m46.717703488s) [seg_pos={0 0}, upload=672894]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Jan 6 16:34:46.846729380 (769h5m46.717812415s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:46.846734600 (769h5m46.717817635s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:47.079572108 (769h5m46.950655143s) [upload=672894]: Done uploading segments...
Jan 6 16:34:47.079590854 (769h5m46.950673889s) [upload=672894]: Upload finished. err: <nil>
Duration: 11.257574211s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....X......................X......X.........................] duration:
16.942750731s trace:
https://jaeger.dev.storj.io/trace/d5e2d8b8127d900
FAIL (2026-01-06 16:39:17 UTC)
orbiter warning: context deadline exceeded
Duration: 16.942772343s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
12 out of 60 is failed (threshold=3) [.....X......X..X.X.X........X.....XX......X..........XX...X.] duration:
39.951848776s trace:
https://jaeger.dev.storj.io/trace/6985ae600f788bdc
FAIL (2026-01-06 16:39:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2, Piece ID: 7VZFPVMIHBNV47544IM26JXIQJP7YYORK7SH7K2JBG3RFJHJGWNA) context deadline exceeded; piecestore: (Node ID: 12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, Piece ID: AYNGFPHKKZ2IYS7LCG6472GWCDEDDOYUYRWXP3NMMIMOMSL7LXBA) context deadline exceeded; piecestore: (Node ID: 12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, Piece ID: 5ZGJR22SDDXZZYU22QGWNHE6JJJP5NMKXBCVK5FT4I4W5LNFXT6A) context deadline exceeded
======LOGS======
34]: Uploading piece...
Jan 6 16:37:29.760860613 (769h8m29.631943648s) [noise=true , address=145.79.227.25:4652 , piece_id=PYZ62FVBB7J6KUPJO6DILLS2ZHWFFWJFU3ALQMGUOOZJXY7USDCA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.761162648 (769h8m29.632245683s) [noise=true , address=77.90.31.17:4769 , piece_id=KZ5JVWDYVHG3KJSWKGGIEB4NNOKXNJGODDYIWYRSEQFNK5APJ43A, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.761460535 (769h8m29.632543571s) [noise=true , address=145.79.227.43:4184 , piece_id=GLHU3ABO4YJ3ORKDGT25SY57CBTSPY3T5VS4ZRMQSNNQ2TR7WINA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.761781757 (769h8m29.632864792s) [noise=true , address=77.90.31.12:4601 , piece_id=7VZFPVMIHBNV47544IM26JXIQJP7YYORK7SH7K2JBG3RFJHJGWNA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.762180437 (769h8m29.633263472s) [noise=true , address=64.49.13.155:4915 , piece_id=DKXB3PYLYIUIA4A2H5AF6ORFJQEMFWGXJWOMD7XRAMQZVGXL4EOA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.762566281 (769h8m29.633649326s) [noise=true , address=77.90.31.14:4669 , piece_id=AYNGFPHKKZ2IYS7LCG6472GWCDEDDOYUYRWXP3NMMIMOMSL7LXBA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.763327480 (769h8m29.634410525s) [noise=true , address=64.49.13.169:4706 , piece_id=6YIXXSB6FXELT5RIOMGJFBIHNQXRUQEE3P3XBEFRRBBBORODP6QQ, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.763642820 (769h8m29.634725855s) [noise=true , address=45.91.116.13:4170 , piece_id=J4RR3FYSK7QBP2CZDZ723YGMZBGQCGJEFCWQ6VMYEV5CXRINI3OQ, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.763959293 (769h8m29.635042328s) [noise=true , address=145.79.227.33:4889 , piece_id=V5NHXRS723RDLNSBCYOLBJZZMU27BLWV6JVDCJOP5MO22UTA7IOQ, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.764227504 (769h8m29.635310539s) [noise=true , address=145.79.227.6:4283 , piece_id=SO753P4LAOA5HVWUEOUQITLHGHGVSH7N47J7QYWW5REPVJJVOUGQ, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.764705895 (769h8m29.63578894s) [noise=true , address=145.79.227.4:4199 , piece_id=D3ATF3UVUW7NDRUR5VGX5VNFRGVDBUP7GDVQ3TE7NLQ7AT2SXE4A, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.765115796 (769h8m29.636198831s) [noise=true , address=145.79.227.40:4070 , piece_id=TTCARB5TEKB66J4OQFPNFKPGH2E7QJPY6MQTVMPGMFENUGFRQ2PA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.765455001 (769h8m29.636538036s) [noise=true , address=145.79.227.19:4107 , piece_id=L5GZHMNTZL3ZQSN3IXBKDNGGSDID2RZPKJVPJT3DZPWB5TMZ7IQQ, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:31.177004929 (769h8m31.048087974s) [upload=672934]: Next returned nil segment
Jan 6 16:37:31.177018294 (769h8m31.04810134s) [upload=672934]: Waiting for error group managing segments...
Jan 6 16:37:31.629873170 (769h8m31.500956205s) [noise=true , address=145.79.227.19:4107 , piece_id=L5GZHMNTZL3ZQSN3IXBKDNGGSDID2RZPKJVPJT3DZPWB5TMZ7IQQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:31.660937181 (769h8m31.532020226s) [noise=true , address=145.79.227.11:4733 , piece_id=NW7A5MKY42A7K7IT37UE2UT5JD3KKR6YEK7KYT54MXI5RCDA6OQA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:31.673141029 (769h8m31.544224074s) [noise=true , address=145.79.227.18:4013 , piece_id=ULB7OJRS4ENT5R6QSADJJHTK7KSXQSBKB2JBQL3SE3SVEWDYNZKQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.002965115 (769h8m31.87404816s) [noise=true , address=77.90.31.14:4669 , piece_id=AYNGFPHKKZ2IYS7LCG6472GWCDEDDOYUYRWXP3NMMIMOMSL7LXBA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.151909408 (769h8m32.022992453s) [noise=true , address=45.91.116.11:4133 , piece_id=5ZGJR22SDDXZZYU22QGWNHE6JJJP5NMKXBCVK5FT4I4W5LNFXT6A, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.563183958 (769h8m32.434266993s) [noise=true , address=145.79.227.43:4184 , piece_id=GLHU3ABO4YJ3ORKDGT25SY57CBTSPY3T5VS4ZRMQSNNQ2TR7WINA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.604451158 (769h8m32.475534193s) [noise=true , address=145.79.227.30:4779 , piece_id=C6ZU63ZIKRQ62T7LIEYG4J22LV7BTA67A4QE4DBU7RJRPMCDN52A, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.668621037 (769h8m32.539704072s) [noise=true , address=145.79.227.30:4765 , piece_id=VOMTCWZBKXLW4VRSOO33CY567X5NPVW6J6WWDEXMMRQCN4LYBPQQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.733299665 (769h8m32.6043827s) [noise=true , address=145.79.227.40:4070 , piece_id=TTCARB5TEKB66J4OQFPNFKPGH2E7QJPY6MQTVMPGMFENUGFRQ2PA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.740288532 (769h8m32.611371567s) [noise=true , address=77.90.31.12:4601 , piece_id=7VZFPVMIHBNV47544IM26JXIQJP7YYORK7SH7K2JBG3RFJHJGWNA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.857976147 (769h8m32.729059192s) [noise=true , address=145.79.227.33:4889 , piece_id=V5NHXRS723RDLNSBCYOLBJZZMU27BLWV6JVDCJOP5MO22UTA7IOQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:33.103891368 (769h8m32.974974403s) [noise=true , address=145.79.227.25:4652 , piece_id=PYZ62FVBB7J6KUPJO6DILLS2ZHWFFWJFU3ALQMGUOOZJXY7USDCA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:33.157267243 (769h8m33.028350288s) [noise=true , address=64.49.13.155:4915 , piece_id=DKXB3PYLYIUIA4A2H5AF6ORFJQEMFWGXJWOMD7XRAMQZVGXL4EOA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:33.327651983 (769h8m33.198735018s) [noise=true , address=77.90.31.17:4769 , piece_id=KZ5JVWDYVHG3KJSWKGGIEB4NNOKXNJGODDYIWYRSEQFNK5APJ43A, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:33.327674626 (769h8m33.198757671s) [seg_pos={0 0}, upload=672934]: Segment reached optimal threshold of 14 pieces.
Jan 6 16:37:33.327850531 (769h8m33.198933566s) [noise=true , address=145.79.227.6:4283 , piece_id=SO753P4LAOA5HVWUEOUQITLHGHGVSH7N47J7QYWW5REPVJJVOUGQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:35736->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:35736->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35736->145.79.227.6:4283: use of closed network connection
Jan 6 16:37:33.327856282 (769h8m33.198939317s) [noise=true , address=145.79.227.6:4283 , piece_id=SO753P4LAOA5HVWUEOUQITLHGHGVSH7N47J7QYWW5REPVJJVOUGQ, seg_pos={0 0}, upload=672934]: Long tail cancellation due to optimal threshold
Jan 6 16:37:33.328024041 (769h8m33.199107086s) [noise=true , address=145.79.227.4:4199 , piece_id=D3ATF3UVUW7NDRUR5VGX5VNFRGVDBUP7GDVQ3TE7NLQ7AT2SXE4A, seg_pos={0 0}, upload=672934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DagQMPpe9F6w5xewXXCfzVBPcmgg6EKCauFsNCWHoHdP56bs7): protocol: write tcp 188.166.225.170:55010->145.79.227.4:4199: use of closed network connection; write tcp 188.166.225.170:55010->145.79.227.4:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55010->145.79.227.4:4199: use of closed network connection
Jan 6 16:37:33.328029502 (769h8m33.199112537s) [noise=true , address=145.79.227.4:4199 , piece_id=D3ATF3UVUW7NDRUR5VGX5VNFRGVDBUP7GDVQ3TE7NLQ7AT2SXE4A, seg_pos={0 0}, upload=672934]: Long tail cancellation due to optimal threshold
Jan 6 16:37:33.328247697 (769h8m33.199330732s) [noise=true , address=64.49.13.169:4706 , piece_id=6YIXXSB6FXELT5RIOMGJFBIHNQXRUQEE3P3XBEFRRBBBORODP6QQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:60742->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:60742->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60742->64.49.13.169:4706: use of closed network connection
Jan 6 16:37:33.328274859 (769h8m33.199357914s) [noise=true , address=64.49.13.169:4706 , piece_id=6YIXXSB6FXELT5RIOMGJFBIHNQXRUQEE3P3XBEFRRBBBORODP6QQ, seg_pos={0 0}, upload=672934]: Long tail cancellation due to optimal threshold
Jan 6 16:37:33.328358257 (769h8m33.199441292s) [noise=true , address=45.91.116.13:4170 , piece_id=J4RR3FYSK7QBP2CZDZ723YGMZBGQCGJEFCWQ6VMYEV5CXRINI3OQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R): protocol: read tcp 188.166.225.170:38244->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 16:37:33.328365240 (769h8m33.199448285s) [noise=true , address=45.91.116.13:4170 , piece_id=J4RR3FYSK7QBP2CZDZ723YGMZBGQCGJEFCWQ6VMYEV5CXRINI3OQ, seg_pos={0 0}, upload=672934]: Long tail cancellation due to optimal threshold
Jan 6 16:37:33.328381261 (769h8m33.199464296s) [seg_pos={0 0}, upload=672934]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 16:37:33.328443740 (769h8m33.199526776s) [upload=672934]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:37:33.328447878 (769h8m33.199530913s) [upload=672934]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:37:33.623102516 (769h8m33.494185551s) [upload=672934]: Done uploading segments...
Jan 6 16:37:33.623147712 (769h8m33.494230757s) [upload=672934]: Upload finished. err: <nil>
Duration: 39.951876128s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [...XX......................X.X....X.X.X....X.X...X....XXX...] duration:
23.447973671s trace:
https://jaeger.dev.storj.io/trace/73dd0c7413e4b545
FAIL (2026-01-06 16:38:54 UTC)
orbiter warning: context deadline exceeded
Duration: 23.448003008s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [......................................................XX....] duration:
16.3443589s trace:
https://jaeger.dev.storj.io/trace/4b3b841c440bfc62
FAIL (2026-01-06 16:38:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913312874 (769h5m40.784395909s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913698277 (769h5m40.784781322s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913862180 (769h5m40.784945215s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914346662 (769h5m40.785429707s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914842456 (769h5m40.785925491s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915307461 (769h5m40.786390506s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915793817 (769h5m40.786876863s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916279973 (769h5m40.787363018s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916759336 (769h5m40.787842371s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.920089677 (769h5m40.791172722s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:41.741185400 (769h5m41.612268445s) [upload=672894]: Next returned nil segment
Jan 6 16:34:41.741201020 (769h5m41.612284055s) [upload=672894]: Waiting for error group managing segments...
Jan 6 16:34:42.136024704 (769h5m42.00710774s) [noise=true , address=185.167.204.205:4036 , piece_id=4XKAIZPEJZAVAS7P2QILKY7XIDIIRIRWH7HTN5OVXUHAL2Y3H4SQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137413950 (769h5m42.008496995s) [noise=true , address=185.167.204.206:4051 , piece_id=XWKJM6ZIHISLSESJ2ZQQQMZTGMK5MHH6L4ZD5RXJ3SMAVASVVVOA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137527506 (769h5m42.008610541s) [noise=true , address=185.167.204.206:4049 , piece_id=4T665GS7XTBVEHUVT6SFIQW4F3TTBPDKZ57PKQHHHWWKC4P4JQPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.138183485 (769h5m42.00926653s) [noise=true , address=185.167.204.206:4048 , piece_id=ZGQYUYRK5PK24UF4PYT67FHNPR4GVFDCJ2OEGRFENLB2GYR2LFVQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.139902609 (769h5m42.010985644s) [noise=true , address=185.167.204.206:4056 , piece_id=KYHDW5RLJTZCURXCFIP7HN3ZYLPPHK7FTO6IIGQ5VJJUPF4ICQXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.140277843 (769h5m42.011360878s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.142394303 (769h5m42.013477348s) [noise=true , address=185.167.204.205:4021 , piece_id=U4X3NZRT7MHG2JTNRVDRYIC72UDMHUQUCY4VFW7JFVLLVACZN7OA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.179529514 (769h5m42.050612549s) [noise=true , address=185.167.204.206:4053 , piece_id=HBFCBJCOLPJLNE4INHP4JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.194387988 (769h5m42.065471023s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.197860059 (769h5m42.068943094s) [noise=true , address=185.167.204.206:4064 , piece_id=JGCXE3QUORVKLRIWEXFZUCT4XUF3DRVA35ZKQU4DQODLZBOH5I4A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.827615690 (769h5m42.698698725s) [noise=true , address=81.171.14.70:4041 , piece_id=NLGMNQJDFISSAHBX3EC6WMWJLP74FMR4MRPUIX4GCCCQRG5RRRCQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.917676689 (769h5m42.788759725s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.932213350 (769h5m42.803296385s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.956051121 (769h5m42.827134196s) [noise=true , address=81.171.14.70:4031 , piece_id=OJKJHNVWTPDITZC5CTQNB7UFPIVZDPE54ZXS4LWRM3XHYAB6443A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.005259852 (769h5m42.876342887s) [noise=true , address=185.167.204.205:4019 , piece_id=I7OP7AQS2DC3C7GZWG5RVJSLDKAZD2434WMBPRYMZFVAJYJG3JHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.052708792 (769h5m42.923791837s) [noise=true , address=81.171.14.70:4046 , piece_id=OV3PMCD6UIFNSQNTCOS3GEQC4676KFJOUH5ZX576OFOYC5EXCZPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.639749858 (769h5m43.510832903s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.719270598 (769h5m43.590353643s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.892052420 (769h5m43.763135466s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.901787498 (769h5m43.772870533s) [noise=true , address=81.171.14.70:4039 , piece_id=VFXOSS5DRL43RDJGPLSH3667HWKAF5ZNC2CF5BA7TDR4NSCG75HA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.019887208 (769h5m43.890970243s) [noise=true , address=185.167.204.206:4050 , piece_id=KSCVNGXUGFV7XRNU5MFIKLLQGZ5BNHTIXQLVHWPMITZOQUA23DGQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.510337921 (769h5m44.381420957s) [noise=true , address=185.167.204.205:4028 , piece_id=QPB4EPPEENW277S4SBIOCWWMBLIE6GKCGEM3TV3MXN6SYFMQAI7Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.692198242 (769h5m44.563281277s) [noise=true , address=89.149.201.147:4010 , piece_id=C57ORVBIZE4QUWCVG3OVG3KW2Z2GYVWUXEOYG5R53KV2WUQLR7JQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.394574949 (769h5m46.265657984s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.543864889 (769h5m46.414947934s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846238946 (769h5m46.717321982s) [noise=true , address=185.167.204.205:4006 , piece_id=W3XXEI7MG37VJHHK2CECNKIIHB7WSQABGVB5W5ZGAWSTFQ4NQMEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846258273 (769h5m46.717341308s) [seg_pos={0 0}, upload=672894]: Segment reached optimal threshold of 26 pieces.
Jan 6 16:34:46.846448986 (769h5m46.717532021s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846454998 (769h5m46.717538033s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846579175 (769h5m46.71766221s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846595475 (769h5m46.71767851s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846620453 (769h5m46.717703488s) [seg_pos={0 0}, upload=672894]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Jan 6 16:34:46.846729380 (769h5m46.717812415s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:46.846734600 (769h5m46.717817635s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:47.079572108 (769h5m46.950655143s) [upload=672894]: Done uploading segments...
Jan 6 16:34:47.079590854 (769h5m46.950673889s) [upload=672894]: Upload finished. err: <nil>
Duration: 16.344387535s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X...X..X.XXXX..X.....XXX........X.......X..XX.....XX.X...X.] duration:
6.243584385s trace:
https://jaeger.dev.storj.io/trace/15a28c27e96f673b
FAIL (2026-01-06 16:38:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
stItem_ObjectBegin
Jan 6 16:37:28.473274302 (769h8m28.344357337s) [upload=672932]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 16:37:28.721840579 (769h8m28.592923625s) [seg_pos={0 0}, upload=672932]: Begin upload segment...
Jan 6 16:37:28.721914441 (769h8m28.592997486s) [seg_pos={0 0}, upload=672932]: Done begin upload segment.
Jan 6 16:37:28.721972401 (769h8m28.593055446s) [noise=true , address=145.79.227.28:4725 , piece_id=P7AQCSLA6GHJIOFCVQCC6CYMYMNRUQUJZ4EF2INABKTOWGXOJXWQ, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.722035762 (769h8m28.593118797s) [noise=true , address=145.79.227.2:4114 , piece_id=R54XVI26BM2WHGPZJ66WRX6C3BU77UZHTFPPDNJERF6KXCTYZU2A, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.722691861 (769h8m28.593774896s) [noise=true , address=145.79.227.15:4874 , piece_id=5RVJMEMS25D2PQP7IFZBXVOFMED3BMAWEDIL33NENI7DVQXWDDWA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.722965723 (769h8m28.594048758s) [noise=true , address=145.79.227.31:4799 , piece_id=JWBIHZUT3EIKXFO6BSRFVVT3JIP437HSVVSGFCCC4LDIEMYEY5QQ, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.723445085 (769h8m28.59452813s) [noise=true , address=77.90.31.19:4841 , piece_id=LFDY56G7H7TT36CTQ2D7CDTRXLKLTAZCMRW6I4PMMEQZKYNSLSTA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.723924528 (769h8m28.595007573s) [noise=true , address=145.79.227.25:4619 , piece_id=R3Q24BOABHJI3G57FDRGJONT2B56PQLFPYSLSNZMPCPT3QGRDA2Q, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.724398911 (769h8m28.595481957s) [noise=true , address=145.79.227.51:4409 , piece_id=SMHE6AEAUKGYSTNTNZQDXD72KLSOQR6U7PSOZSDJOVOYWSIB2VEA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.724785818 (769h8m28.595868853s) [noise=true , address=145.79.227.13:4802 , piece_id=V2XTKHCPX7V654YGWDKVLJ3ZZHQW3TIMBNQAV6BIOIFXZUMGEB7Q, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.725201720 (769h8m28.596284755s) [noise=true , address=145.79.227.14:4839 , piece_id=NT3CMTRY2FTJPB4GS3HWQS7FX4HDENRQG3IXRBYRHPJL67TSBJEA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.725559291 (769h8m28.596642336s) [noise=true , address=145.79.227.47:4266 , piece_id=GMIEPW67HDNRJZTHCWN5TUCQDW4WRBDTSWRY6KGG33VLNHHMUQVA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.726013656 (769h8m28.597096691s) [noise=true , address=145.79.227.41:4087 , piece_id=QSYXOB7S5RFC6VOCQBJDKJSXE46DBDEGMDULHNH4DXOEIUWNI74A, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.727455210 (769h8m28.598538255s) [noise=true , address=77.90.31.11:4553 , piece_id=QL3IPBKRC2G4PSPKNE6QWIUCEIDUEL3L5EW6RGDJQ5FMGSGGAGIQ, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.729102628 (769h8m28.600185673s) [noise=true , address=45.91.116.12:4225 , piece_id=HYNGZEYFA2DGUPEZ2MK4ZK4NWDIUNMICRVND43QV72XMCNI4I3BA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.731357110 (769h8m28.602440155s) [noise=true , address=145.79.227.19:4079 , piece_id=3N32WBSN4AAM65VXOG67RT65DDC3WPJVUWZUSYW7FBHAHQYTREXA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.733220569 (769h8m28.604303614s) [noise=true , address=145.79.227.46:4603 , piece_id=HTFQUA7S5YO7PM2E5MVKZMMFWBRA5ZOUXXEPBRCPZMQANMZU6QPA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.734616186 (769h8m28.605699221s) [noise=true , address=45.91.116.10:4084 , piece_id=2CCD33ZBO43ZTBR7LGWKO3KHM3NWBOTJYXLPADJLZYR4WNUHM5BQ, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:30.434134861 (769h8m30.305217896s) [upload=672932]: Next returned nil segment
Jan 6 16:37:30.434148446 (769h8m30.305231491s) [upload=672932]: Waiting for error group managing segments...
Jan 6 16:37:30.731706634 (769h8m30.602789679s) [noise=true , address=145.79.227.31:4799 , piece_id=JWBIHZUT3EIKXFO6BSRFVVT3JIP437HSVVSGFCCC4LDIEMYEY5QQ, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.734574354 (769h8m30.605657399s) [noise=true , address=145.79.227.14:4839 , piece_id=NT3CMTRY2FTJPB4GS3HWQS7FX4HDENRQG3IXRBYRHPJL67TSBJEA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.736418766 (769h8m30.607501811s) [noise=true , address=145.79.227.25:4619 , piece_id=R3Q24BOABHJI3G57FDRGJONT2B56PQLFPYSLSNZMPCPT3QGRDA2Q, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.744377039 (769h8m30.615460084s) [noise=true , address=145.79.227.15:4874 , piece_id=5RVJMEMS25D2PQP7IFZBXVOFMED3BMAWEDIL33NENI7DVQXWDDWA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.746084300 (769h8m30.617167345s) [noise=true , address=145.79.227.19:4079 , piece_id=3N32WBSN4AAM65VXOG67RT65DDC3WPJVUWZUSYW7FBHAHQYTREXA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.785740763 (769h8m30.656823808s) [noise=true , address=145.79.227.2:4114 , piece_id=R54XVI26BM2WHGPZJ66WRX6C3BU77UZHTFPPDNJERF6KXCTYZU2A, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.789966338 (769h8m30.661049384s) [noise=true , address=145.79.227.28:4725 , piece_id=P7AQCSLA6GHJIOFCVQCC6CYMYMNRUQUJZ4EF2INABKTOWGXOJXWQ, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.799913760 (769h8m30.670996795s) [noise=true , address=145.79.227.13:4802 , piece_id=V2XTKHCPX7V654YGWDKVLJ3ZZHQW3TIMBNQAV6BIOIFXZUMGEB7Q, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.817666644 (769h8m30.688749679s) [noise=true , address=145.79.227.47:4266 , piece_id=GMIEPW67HDNRJZTHCWN5TUCQDW4WRBDTSWRY6KGG33VLNHHMUQVA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.829374427 (769h8m30.700457472s) [noise=true , address=145.79.227.51:4409 , piece_id=SMHE6AEAUKGYSTNTNZQDXD72KLSOQR6U7PSOZSDJOVOYWSIB2VEA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:31.072460932 (769h8m30.943543967s) [noise=true , address=77.90.31.19:4841 , piece_id=LFDY56G7H7TT36CTQ2D7CDTRXLKLTAZCMRW6I4PMMEQZKYNSLSTA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:31.190755042 (769h8m31.061838087s) [noise=true , address=45.91.116.10:4084 , piece_id=2CCD33ZBO43ZTBR7LGWKO3KHM3NWBOTJYXLPADJLZYR4WNUHM5BQ, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:31.204591468 (769h8m31.075674504s) [noise=true , address=45.91.116.12:4225 , piece_id=HYNGZEYFA2DGUPEZ2MK4ZK4NWDIUNMICRVND43QV72XMCNI4I3BA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:31.204610946 (769h8m31.075693991s) [seg_pos={0 0}, upload=672932]: Segment reached optimal threshold of 13 pieces.
Jan 6 16:37:31.204860461 (769h8m31.075943506s) [noise=true , address=145.79.227.46:4603 , piece_id=HTFQUA7S5YO7PM2E5MVKZMMFWBRA5ZOUXXEPBRCPZMQANMZU6QPA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: write tcp 188.166.225.170:39256->145.79.227.46:4603: use of closed network connection; write tcp 188.166.225.170:39256->145.79.227.46:4603: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39256->145.79.227.46:4603: use of closed network connection
Jan 6 16:37:31.204867574 (769h8m31.075950609s) [noise=true , address=145.79.227.46:4603 , piece_id=HTFQUA7S5YO7PM2E5MVKZMMFWBRA5ZOUXXEPBRCPZMQANMZU6QPA, seg_pos={0 0}, upload=672932]: Long tail cancellation due to optimal threshold
Jan 6 16:37:31.204918652 (769h8m31.076001687s) [noise=true , address=77.90.31.11:4553 , piece_id=QL3IPBKRC2G4PSPKNE6QWIUCEIDUEL3L5EW6RGDJQ5FMGSGGAGIQ, seg_pos={0 0}, upload=672932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:44946->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:44946->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44946->77.90.31.11:4553: use of closed network connection
Jan 6 16:37:31.204957346 (769h8m31.076040381s) [noise=true , address=77.90.31.11:4553 , piece_id=QL3IPBKRC2G4PSPKNE6QWIUCEIDUEL3L5EW6RGDJQ5FMGSGGAGIQ, seg_pos={0 0}, upload=672932]: Long tail cancellation due to optimal threshold
Jan 6 16:37:31.205091570 (769h8m31.076174616s) [noise=true , address=145.79.227.41:4087 , piece_id=QSYXOB7S5RFC6VOCQBJDKJSXE46DBDEGMDULHNH4DXOEIUWNI74A, seg_pos={0 0}, upload=672932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:43320->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:43320->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43320->145.79.227.41:4087: use of closed network connection
Jan 6 16:37:31.205098164 (769h8m31.076181199s) [noise=true , address=145.79.227.41:4087 , piece_id=QSYXOB7S5RFC6VOCQBJDKJSXE46DBDEGMDULHNH4DXOEIUWNI74A, seg_pos={0 0}, upload=672932]: Long tail cancellation due to optimal threshold
Jan 6 16:37:31.205118993 (769h8m31.076202028s) [seg_pos={0 0}, upload=672932]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 16:37:31.205199687 (769h8m31.076282722s) [upload=672932]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:37:31.205204727 (769h8m31.076287772s) [upload=672932]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:37:31.501536788 (769h8m31.372619833s) [upload=672932]: Done uploading segments...
Jan 6 16:37:31.501576303 (769h8m31.372659349s) [upload=672932]: Upload finished. err: <nil>
Duration: 6.243618651s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....X......................X......X........................] duration:
14.556500443s trace:
https://jaeger.dev.storj.io/trace/d5e2d8b8127d900
FAIL (2026-01-06 16:38:33 UTC)
orbiter warning: context deadline exceeded
Duration: 14.556525673s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.XX.XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000775175s trace:
https://jaeger.dev.storj.io/trace/4a3d182fe4821aac
FAIL (2026-01-06 16:38:31 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000800374s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
12 out of 60 is failed (threshold=3) [......X......X..X.X.X........X.....XX......X..........XX...X] duration:
1m0.004056939s trace:
https://jaeger.dev.storj.io/trace/6985ae600f788bdc
FAIL (2026-01-06 16:38:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2, Piece ID: 7VZFPVMIHBNV47544IM26JXIQJP7YYORK7SH7K2JBG3RFJHJGWNA) context deadline exceeded; piecestore: (Node ID: 12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, Piece ID: AYNGFPHKKZ2IYS7LCG6472GWCDEDDOYUYRWXP3NMMIMOMSL7LXBA) context deadline exceeded; piecestore: (Node ID: 12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, Piece ID: 5ZGJR22SDDXZZYU22QGWNHE6JJJP5NMKXBCVK5FT4I4W5LNFXT6A) context deadline exceeded
======LOGS======
34]: Uploading piece...
Jan 6 16:37:29.760860613 (769h8m29.631943648s) [noise=true , address=145.79.227.25:4652 , piece_id=PYZ62FVBB7J6KUPJO6DILLS2ZHWFFWJFU3ALQMGUOOZJXY7USDCA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.761162648 (769h8m29.632245683s) [noise=true , address=77.90.31.17:4769 , piece_id=KZ5JVWDYVHG3KJSWKGGIEB4NNOKXNJGODDYIWYRSEQFNK5APJ43A, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.761460535 (769h8m29.632543571s) [noise=true , address=145.79.227.43:4184 , piece_id=GLHU3ABO4YJ3ORKDGT25SY57CBTSPY3T5VS4ZRMQSNNQ2TR7WINA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.761781757 (769h8m29.632864792s) [noise=true , address=77.90.31.12:4601 , piece_id=7VZFPVMIHBNV47544IM26JXIQJP7YYORK7SH7K2JBG3RFJHJGWNA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.762180437 (769h8m29.633263472s) [noise=true , address=64.49.13.155:4915 , piece_id=DKXB3PYLYIUIA4A2H5AF6ORFJQEMFWGXJWOMD7XRAMQZVGXL4EOA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.762566281 (769h8m29.633649326s) [noise=true , address=77.90.31.14:4669 , piece_id=AYNGFPHKKZ2IYS7LCG6472GWCDEDDOYUYRWXP3NMMIMOMSL7LXBA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.763327480 (769h8m29.634410525s) [noise=true , address=64.49.13.169:4706 , piece_id=6YIXXSB6FXELT5RIOMGJFBIHNQXRUQEE3P3XBEFRRBBBORODP6QQ, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.763642820 (769h8m29.634725855s) [noise=true , address=45.91.116.13:4170 , piece_id=J4RR3FYSK7QBP2CZDZ723YGMZBGQCGJEFCWQ6VMYEV5CXRINI3OQ, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.763959293 (769h8m29.635042328s) [noise=true , address=145.79.227.33:4889 , piece_id=V5NHXRS723RDLNSBCYOLBJZZMU27BLWV6JVDCJOP5MO22UTA7IOQ, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.764227504 (769h8m29.635310539s) [noise=true , address=145.79.227.6:4283 , piece_id=SO753P4LAOA5HVWUEOUQITLHGHGVSH7N47J7QYWW5REPVJJVOUGQ, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.764705895 (769h8m29.63578894s) [noise=true , address=145.79.227.4:4199 , piece_id=D3ATF3UVUW7NDRUR5VGX5VNFRGVDBUP7GDVQ3TE7NLQ7AT2SXE4A, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.765115796 (769h8m29.636198831s) [noise=true , address=145.79.227.40:4070 , piece_id=TTCARB5TEKB66J4OQFPNFKPGH2E7QJPY6MQTVMPGMFENUGFRQ2PA, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:29.765455001 (769h8m29.636538036s) [noise=true , address=145.79.227.19:4107 , piece_id=L5GZHMNTZL3ZQSN3IXBKDNGGSDID2RZPKJVPJT3DZPWB5TMZ7IQQ, seg_pos={0 0}, upload=672934]: Uploading piece...
Jan 6 16:37:31.177004929 (769h8m31.048087974s) [upload=672934]: Next returned nil segment
Jan 6 16:37:31.177018294 (769h8m31.04810134s) [upload=672934]: Waiting for error group managing segments...
Jan 6 16:37:31.629873170 (769h8m31.500956205s) [noise=true , address=145.79.227.19:4107 , piece_id=L5GZHMNTZL3ZQSN3IXBKDNGGSDID2RZPKJVPJT3DZPWB5TMZ7IQQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:31.660937181 (769h8m31.532020226s) [noise=true , address=145.79.227.11:4733 , piece_id=NW7A5MKY42A7K7IT37UE2UT5JD3KKR6YEK7KYT54MXI5RCDA6OQA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:31.673141029 (769h8m31.544224074s) [noise=true , address=145.79.227.18:4013 , piece_id=ULB7OJRS4ENT5R6QSADJJHTK7KSXQSBKB2JBQL3SE3SVEWDYNZKQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.002965115 (769h8m31.87404816s) [noise=true , address=77.90.31.14:4669 , piece_id=AYNGFPHKKZ2IYS7LCG6472GWCDEDDOYUYRWXP3NMMIMOMSL7LXBA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.151909408 (769h8m32.022992453s) [noise=true , address=45.91.116.11:4133 , piece_id=5ZGJR22SDDXZZYU22QGWNHE6JJJP5NMKXBCVK5FT4I4W5LNFXT6A, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.563183958 (769h8m32.434266993s) [noise=true , address=145.79.227.43:4184 , piece_id=GLHU3ABO4YJ3ORKDGT25SY57CBTSPY3T5VS4ZRMQSNNQ2TR7WINA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.604451158 (769h8m32.475534193s) [noise=true , address=145.79.227.30:4779 , piece_id=C6ZU63ZIKRQ62T7LIEYG4J22LV7BTA67A4QE4DBU7RJRPMCDN52A, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.668621037 (769h8m32.539704072s) [noise=true , address=145.79.227.30:4765 , piece_id=VOMTCWZBKXLW4VRSOO33CY567X5NPVW6J6WWDEXMMRQCN4LYBPQQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.733299665 (769h8m32.6043827s) [noise=true , address=145.79.227.40:4070 , piece_id=TTCARB5TEKB66J4OQFPNFKPGH2E7QJPY6MQTVMPGMFENUGFRQ2PA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.740288532 (769h8m32.611371567s) [noise=true , address=77.90.31.12:4601 , piece_id=7VZFPVMIHBNV47544IM26JXIQJP7YYORK7SH7K2JBG3RFJHJGWNA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:32.857976147 (769h8m32.729059192s) [noise=true , address=145.79.227.33:4889 , piece_id=V5NHXRS723RDLNSBCYOLBJZZMU27BLWV6JVDCJOP5MO22UTA7IOQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:33.103891368 (769h8m32.974974403s) [noise=true , address=145.79.227.25:4652 , piece_id=PYZ62FVBB7J6KUPJO6DILLS2ZHWFFWJFU3ALQMGUOOZJXY7USDCA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:33.157267243 (769h8m33.028350288s) [noise=true , address=64.49.13.155:4915 , piece_id=DKXB3PYLYIUIA4A2H5AF6ORFJQEMFWGXJWOMD7XRAMQZVGXL4EOA, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:33.327651983 (769h8m33.198735018s) [noise=true , address=77.90.31.17:4769 , piece_id=KZ5JVWDYVHG3KJSWKGGIEB4NNOKXNJGODDYIWYRSEQFNK5APJ43A, seg_pos={0 0}, upload=672934]: Done uploading piece. err: <nil>
Jan 6 16:37:33.327674626 (769h8m33.198757671s) [seg_pos={0 0}, upload=672934]: Segment reached optimal threshold of 14 pieces.
Jan 6 16:37:33.327850531 (769h8m33.198933566s) [noise=true , address=145.79.227.6:4283 , piece_id=SO753P4LAOA5HVWUEOUQITLHGHGVSH7N47J7QYWW5REPVJJVOUGQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:35736->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:35736->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35736->145.79.227.6:4283: use of closed network connection
Jan 6 16:37:33.327856282 (769h8m33.198939317s) [noise=true , address=145.79.227.6:4283 , piece_id=SO753P4LAOA5HVWUEOUQITLHGHGVSH7N47J7QYWW5REPVJJVOUGQ, seg_pos={0 0}, upload=672934]: Long tail cancellation due to optimal threshold
Jan 6 16:37:33.328024041 (769h8m33.199107086s) [noise=true , address=145.79.227.4:4199 , piece_id=D3ATF3UVUW7NDRUR5VGX5VNFRGVDBUP7GDVQ3TE7NLQ7AT2SXE4A, seg_pos={0 0}, upload=672934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DagQMPpe9F6w5xewXXCfzVBPcmgg6EKCauFsNCWHoHdP56bs7): protocol: write tcp 188.166.225.170:55010->145.79.227.4:4199: use of closed network connection; write tcp 188.166.225.170:55010->145.79.227.4:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55010->145.79.227.4:4199: use of closed network connection
Jan 6 16:37:33.328029502 (769h8m33.199112537s) [noise=true , address=145.79.227.4:4199 , piece_id=D3ATF3UVUW7NDRUR5VGX5VNFRGVDBUP7GDVQ3TE7NLQ7AT2SXE4A, seg_pos={0 0}, upload=672934]: Long tail cancellation due to optimal threshold
Jan 6 16:37:33.328247697 (769h8m33.199330732s) [noise=true , address=64.49.13.169:4706 , piece_id=6YIXXSB6FXELT5RIOMGJFBIHNQXRUQEE3P3XBEFRRBBBORODP6QQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:60742->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:60742->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60742->64.49.13.169:4706: use of closed network connection
Jan 6 16:37:33.328274859 (769h8m33.199357914s) [noise=true , address=64.49.13.169:4706 , piece_id=6YIXXSB6FXELT5RIOMGJFBIHNQXRUQEE3P3XBEFRRBBBORODP6QQ, seg_pos={0 0}, upload=672934]: Long tail cancellation due to optimal threshold
Jan 6 16:37:33.328358257 (769h8m33.199441292s) [noise=true , address=45.91.116.13:4170 , piece_id=J4RR3FYSK7QBP2CZDZ723YGMZBGQCGJEFCWQ6VMYEV5CXRINI3OQ, seg_pos={0 0}, upload=672934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R): protocol: read tcp 188.166.225.170:38244->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 16:37:33.328365240 (769h8m33.199448285s) [noise=true , address=45.91.116.13:4170 , piece_id=J4RR3FYSK7QBP2CZDZ723YGMZBGQCGJEFCWQ6VMYEV5CXRINI3OQ, seg_pos={0 0}, upload=672934]: Long tail cancellation due to optimal threshold
Jan 6 16:37:33.328381261 (769h8m33.199464296s) [seg_pos={0 0}, upload=672934]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 16:37:33.328443740 (769h8m33.199526776s) [upload=672934]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:37:33.328447878 (769h8m33.199530913s) [upload=672934]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:37:33.623102516 (769h8m33.494185551s) [upload=672934]: Done uploading segments...
Jan 6 16:37:33.623147712 (769h8m33.494230757s) [upload=672934]: Upload finished. err: <nil>
Duration: 1m0.004069693s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [X.....X......................X......X.......................] duration:
18.926585331s trace:
https://jaeger.dev.storj.io/trace/d5e2d8b8127d900
FAIL (2026-01-06 16:38:01 UTC)
orbiter warning: context deadline exceeded
Duration: 18.926614415s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX.XX.XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration:
5.002139984s trace:
https://jaeger.dev.storj.io/trace/2e76728ac7c49ab3
FAIL (2026-01-06 16:38:00 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002162588s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [..X...X..X.XXXX..X.....XXX........X.......X..XX.....XX.X...X] duration:
30.003883979s trace:
https://jaeger.dev.storj.io/trace/15a28c27e96f673b
FAIL (2026-01-06 16:37:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
stItem_ObjectBegin
Jan 6 16:37:28.473274302 (769h8m28.344357337s) [upload=672932]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 16:37:28.721840579 (769h8m28.592923625s) [seg_pos={0 0}, upload=672932]: Begin upload segment...
Jan 6 16:37:28.721914441 (769h8m28.592997486s) [seg_pos={0 0}, upload=672932]: Done begin upload segment.
Jan 6 16:37:28.721972401 (769h8m28.593055446s) [noise=true , address=145.79.227.28:4725 , piece_id=P7AQCSLA6GHJIOFCVQCC6CYMYMNRUQUJZ4EF2INABKTOWGXOJXWQ, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.722035762 (769h8m28.593118797s) [noise=true , address=145.79.227.2:4114 , piece_id=R54XVI26BM2WHGPZJ66WRX6C3BU77UZHTFPPDNJERF6KXCTYZU2A, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.722691861 (769h8m28.593774896s) [noise=true , address=145.79.227.15:4874 , piece_id=5RVJMEMS25D2PQP7IFZBXVOFMED3BMAWEDIL33NENI7DVQXWDDWA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.722965723 (769h8m28.594048758s) [noise=true , address=145.79.227.31:4799 , piece_id=JWBIHZUT3EIKXFO6BSRFVVT3JIP437HSVVSGFCCC4LDIEMYEY5QQ, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.723445085 (769h8m28.59452813s) [noise=true , address=77.90.31.19:4841 , piece_id=LFDY56G7H7TT36CTQ2D7CDTRXLKLTAZCMRW6I4PMMEQZKYNSLSTA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.723924528 (769h8m28.595007573s) [noise=true , address=145.79.227.25:4619 , piece_id=R3Q24BOABHJI3G57FDRGJONT2B56PQLFPYSLSNZMPCPT3QGRDA2Q, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.724398911 (769h8m28.595481957s) [noise=true , address=145.79.227.51:4409 , piece_id=SMHE6AEAUKGYSTNTNZQDXD72KLSOQR6U7PSOZSDJOVOYWSIB2VEA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.724785818 (769h8m28.595868853s) [noise=true , address=145.79.227.13:4802 , piece_id=V2XTKHCPX7V654YGWDKVLJ3ZZHQW3TIMBNQAV6BIOIFXZUMGEB7Q, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.725201720 (769h8m28.596284755s) [noise=true , address=145.79.227.14:4839 , piece_id=NT3CMTRY2FTJPB4GS3HWQS7FX4HDENRQG3IXRBYRHPJL67TSBJEA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.725559291 (769h8m28.596642336s) [noise=true , address=145.79.227.47:4266 , piece_id=GMIEPW67HDNRJZTHCWN5TUCQDW4WRBDTSWRY6KGG33VLNHHMUQVA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.726013656 (769h8m28.597096691s) [noise=true , address=145.79.227.41:4087 , piece_id=QSYXOB7S5RFC6VOCQBJDKJSXE46DBDEGMDULHNH4DXOEIUWNI74A, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.727455210 (769h8m28.598538255s) [noise=true , address=77.90.31.11:4553 , piece_id=QL3IPBKRC2G4PSPKNE6QWIUCEIDUEL3L5EW6RGDJQ5FMGSGGAGIQ, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.729102628 (769h8m28.600185673s) [noise=true , address=45.91.116.12:4225 , piece_id=HYNGZEYFA2DGUPEZ2MK4ZK4NWDIUNMICRVND43QV72XMCNI4I3BA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.731357110 (769h8m28.602440155s) [noise=true , address=145.79.227.19:4079 , piece_id=3N32WBSN4AAM65VXOG67RT65DDC3WPJVUWZUSYW7FBHAHQYTREXA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.733220569 (769h8m28.604303614s) [noise=true , address=145.79.227.46:4603 , piece_id=HTFQUA7S5YO7PM2E5MVKZMMFWBRA5ZOUXXEPBRCPZMQANMZU6QPA, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:28.734616186 (769h8m28.605699221s) [noise=true , address=45.91.116.10:4084 , piece_id=2CCD33ZBO43ZTBR7LGWKO3KHM3NWBOTJYXLPADJLZYR4WNUHM5BQ, seg_pos={0 0}, upload=672932]: Uploading piece...
Jan 6 16:37:30.434134861 (769h8m30.305217896s) [upload=672932]: Next returned nil segment
Jan 6 16:37:30.434148446 (769h8m30.305231491s) [upload=672932]: Waiting for error group managing segments...
Jan 6 16:37:30.731706634 (769h8m30.602789679s) [noise=true , address=145.79.227.31:4799 , piece_id=JWBIHZUT3EIKXFO6BSRFVVT3JIP437HSVVSGFCCC4LDIEMYEY5QQ, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.734574354 (769h8m30.605657399s) [noise=true , address=145.79.227.14:4839 , piece_id=NT3CMTRY2FTJPB4GS3HWQS7FX4HDENRQG3IXRBYRHPJL67TSBJEA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.736418766 (769h8m30.607501811s) [noise=true , address=145.79.227.25:4619 , piece_id=R3Q24BOABHJI3G57FDRGJONT2B56PQLFPYSLSNZMPCPT3QGRDA2Q, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.744377039 (769h8m30.615460084s) [noise=true , address=145.79.227.15:4874 , piece_id=5RVJMEMS25D2PQP7IFZBXVOFMED3BMAWEDIL33NENI7DVQXWDDWA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.746084300 (769h8m30.617167345s) [noise=true , address=145.79.227.19:4079 , piece_id=3N32WBSN4AAM65VXOG67RT65DDC3WPJVUWZUSYW7FBHAHQYTREXA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.785740763 (769h8m30.656823808s) [noise=true , address=145.79.227.2:4114 , piece_id=R54XVI26BM2WHGPZJ66WRX6C3BU77UZHTFPPDNJERF6KXCTYZU2A, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.789966338 (769h8m30.661049384s) [noise=true , address=145.79.227.28:4725 , piece_id=P7AQCSLA6GHJIOFCVQCC6CYMYMNRUQUJZ4EF2INABKTOWGXOJXWQ, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.799913760 (769h8m30.670996795s) [noise=true , address=145.79.227.13:4802 , piece_id=V2XTKHCPX7V654YGWDKVLJ3ZZHQW3TIMBNQAV6BIOIFXZUMGEB7Q, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.817666644 (769h8m30.688749679s) [noise=true , address=145.79.227.47:4266 , piece_id=GMIEPW67HDNRJZTHCWN5TUCQDW4WRBDTSWRY6KGG33VLNHHMUQVA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:30.829374427 (769h8m30.700457472s) [noise=true , address=145.79.227.51:4409 , piece_id=SMHE6AEAUKGYSTNTNZQDXD72KLSOQR6U7PSOZSDJOVOYWSIB2VEA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:31.072460932 (769h8m30.943543967s) [noise=true , address=77.90.31.19:4841 , piece_id=LFDY56G7H7TT36CTQ2D7CDTRXLKLTAZCMRW6I4PMMEQZKYNSLSTA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:31.190755042 (769h8m31.061838087s) [noise=true , address=45.91.116.10:4084 , piece_id=2CCD33ZBO43ZTBR7LGWKO3KHM3NWBOTJYXLPADJLZYR4WNUHM5BQ, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:31.204591468 (769h8m31.075674504s) [noise=true , address=45.91.116.12:4225 , piece_id=HYNGZEYFA2DGUPEZ2MK4ZK4NWDIUNMICRVND43QV72XMCNI4I3BA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: <nil>
Jan 6 16:37:31.204610946 (769h8m31.075693991s) [seg_pos={0 0}, upload=672932]: Segment reached optimal threshold of 13 pieces.
Jan 6 16:37:31.204860461 (769h8m31.075943506s) [noise=true , address=145.79.227.46:4603 , piece_id=HTFQUA7S5YO7PM2E5MVKZMMFWBRA5ZOUXXEPBRCPZMQANMZU6QPA, seg_pos={0 0}, upload=672932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: write tcp 188.166.225.170:39256->145.79.227.46:4603: use of closed network connection; write tcp 188.166.225.170:39256->145.79.227.46:4603: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39256->145.79.227.46:4603: use of closed network connection
Jan 6 16:37:31.204867574 (769h8m31.075950609s) [noise=true , address=145.79.227.46:4603 , piece_id=HTFQUA7S5YO7PM2E5MVKZMMFWBRA5ZOUXXEPBRCPZMQANMZU6QPA, seg_pos={0 0}, upload=672932]: Long tail cancellation due to optimal threshold
Jan 6 16:37:31.204918652 (769h8m31.076001687s) [noise=true , address=77.90.31.11:4553 , piece_id=QL3IPBKRC2G4PSPKNE6QWIUCEIDUEL3L5EW6RGDJQ5FMGSGGAGIQ, seg_pos={0 0}, upload=672932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:44946->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:44946->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44946->77.90.31.11:4553: use of closed network connection
Jan 6 16:37:31.204957346 (769h8m31.076040381s) [noise=true , address=77.90.31.11:4553 , piece_id=QL3IPBKRC2G4PSPKNE6QWIUCEIDUEL3L5EW6RGDJQ5FMGSGGAGIQ, seg_pos={0 0}, upload=672932]: Long tail cancellation due to optimal threshold
Jan 6 16:37:31.205091570 (769h8m31.076174616s) [noise=true , address=145.79.227.41:4087 , piece_id=QSYXOB7S5RFC6VOCQBJDKJSXE46DBDEGMDULHNH4DXOEIUWNI74A, seg_pos={0 0}, upload=672932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:43320->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:43320->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43320->145.79.227.41:4087: use of closed network connection
Jan 6 16:37:31.205098164 (769h8m31.076181199s) [noise=true , address=145.79.227.41:4087 , piece_id=QSYXOB7S5RFC6VOCQBJDKJSXE46DBDEGMDULHNH4DXOEIUWNI74A, seg_pos={0 0}, upload=672932]: Long tail cancellation due to optimal threshold
Jan 6 16:37:31.205118993 (769h8m31.076202028s) [seg_pos={0 0}, upload=672932]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 16:37:31.205199687 (769h8m31.076282722s) [upload=672932]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:37:31.205204727 (769h8m31.076287772s) [upload=672932]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:37:31.501536788 (769h8m31.372619833s) [upload=672932]: Done uploading segments...
Jan 6 16:37:31.501576303 (769h8m31.372659349s) [upload=672932]: Upload finished. err: <nil>
Duration: 30.003903414s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................XX...] duration:
17.56945225s trace:
https://jaeger.dev.storj.io/trace/4b3b841c440bfc62
FAIL (2026-01-06 16:37:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913312874 (769h5m40.784395909s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913698277 (769h5m40.784781322s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913862180 (769h5m40.784945215s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914346662 (769h5m40.785429707s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914842456 (769h5m40.785925491s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915307461 (769h5m40.786390506s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915793817 (769h5m40.786876863s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916279973 (769h5m40.787363018s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916759336 (769h5m40.787842371s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.920089677 (769h5m40.791172722s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:41.741185400 (769h5m41.612268445s) [upload=672894]: Next returned nil segment
Jan 6 16:34:41.741201020 (769h5m41.612284055s) [upload=672894]: Waiting for error group managing segments...
Jan 6 16:34:42.136024704 (769h5m42.00710774s) [noise=true , address=185.167.204.205:4036 , piece_id=4XKAIZPEJZAVAS7P2QILKY7XIDIIRIRWH7HTN5OVXUHAL2Y3H4SQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137413950 (769h5m42.008496995s) [noise=true , address=185.167.204.206:4051 , piece_id=XWKJM6ZIHISLSESJ2ZQQQMZTGMK5MHH6L4ZD5RXJ3SMAVASVVVOA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137527506 (769h5m42.008610541s) [noise=true , address=185.167.204.206:4049 , piece_id=4T665GS7XTBVEHUVT6SFIQW4F3TTBPDKZ57PKQHHHWWKC4P4JQPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.138183485 (769h5m42.00926653s) [noise=true , address=185.167.204.206:4048 , piece_id=ZGQYUYRK5PK24UF4PYT67FHNPR4GVFDCJ2OEGRFENLB2GYR2LFVQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.139902609 (769h5m42.010985644s) [noise=true , address=185.167.204.206:4056 , piece_id=KYHDW5RLJTZCURXCFIP7HN3ZYLPPHK7FTO6IIGQ5VJJUPF4ICQXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.140277843 (769h5m42.011360878s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.142394303 (769h5m42.013477348s) [noise=true , address=185.167.204.205:4021 , piece_id=U4X3NZRT7MHG2JTNRVDRYIC72UDMHUQUCY4VFW7JFVLLVACZN7OA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.179529514 (769h5m42.050612549s) [noise=true , address=185.167.204.206:4053 , piece_id=HBFCBJCOLPJLNE4INHP4JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.194387988 (769h5m42.065471023s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.197860059 (769h5m42.068943094s) [noise=true , address=185.167.204.206:4064 , piece_id=JGCXE3QUORVKLRIWEXFZUCT4XUF3DRVA35ZKQU4DQODLZBOH5I4A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.827615690 (769h5m42.698698725s) [noise=true , address=81.171.14.70:4041 , piece_id=NLGMNQJDFISSAHBX3EC6WMWJLP74FMR4MRPUIX4GCCCQRG5RRRCQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.917676689 (769h5m42.788759725s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.932213350 (769h5m42.803296385s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.956051121 (769h5m42.827134196s) [noise=true , address=81.171.14.70:4031 , piece_id=OJKJHNVWTPDITZC5CTQNB7UFPIVZDPE54ZXS4LWRM3XHYAB6443A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.005259852 (769h5m42.876342887s) [noise=true , address=185.167.204.205:4019 , piece_id=I7OP7AQS2DC3C7GZWG5RVJSLDKAZD2434WMBPRYMZFVAJYJG3JHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.052708792 (769h5m42.923791837s) [noise=true , address=81.171.14.70:4046 , piece_id=OV3PMCD6UIFNSQNTCOS3GEQC4676KFJOUH5ZX576OFOYC5EXCZPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.639749858 (769h5m43.510832903s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.719270598 (769h5m43.590353643s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.892052420 (769h5m43.763135466s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.901787498 (769h5m43.772870533s) [noise=true , address=81.171.14.70:4039 , piece_id=VFXOSS5DRL43RDJGPLSH3667HWKAF5ZNC2CF5BA7TDR4NSCG75HA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.019887208 (769h5m43.890970243s) [noise=true , address=185.167.204.206:4050 , piece_id=KSCVNGXUGFV7XRNU5MFIKLLQGZ5BNHTIXQLVHWPMITZOQUA23DGQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.510337921 (769h5m44.381420957s) [noise=true , address=185.167.204.205:4028 , piece_id=QPB4EPPEENW277S4SBIOCWWMBLIE6GKCGEM3TV3MXN6SYFMQAI7Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.692198242 (769h5m44.563281277s) [noise=true , address=89.149.201.147:4010 , piece_id=C57ORVBIZE4QUWCVG3OVG3KW2Z2GYVWUXEOYG5R53KV2WUQLR7JQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.394574949 (769h5m46.265657984s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.543864889 (769h5m46.414947934s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846238946 (769h5m46.717321982s) [noise=true , address=185.167.204.205:4006 , piece_id=W3XXEI7MG37VJHHK2CECNKIIHB7WSQABGVB5W5ZGAWSTFQ4NQMEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846258273 (769h5m46.717341308s) [seg_pos={0 0}, upload=672894]: Segment reached optimal threshold of 26 pieces.
Jan 6 16:34:46.846448986 (769h5m46.717532021s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846454998 (769h5m46.717538033s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846579175 (769h5m46.71766221s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846595475 (769h5m46.71767851s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846620453 (769h5m46.717703488s) [seg_pos={0 0}, upload=672894]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Jan 6 16:34:46.846729380 (769h5m46.717812415s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:46.846734600 (769h5m46.717817635s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:47.079572108 (769h5m46.950655143s) [upload=672894]: Done uploading segments...
Jan 6 16:34:47.079590854 (769h5m46.950673889s) [upload=672894]: Upload finished. err: <nil>
Duration: 17.569476575s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [....XX......................X.X....X.X.X....X.X...X....XXX..] duration:
11.595087629s trace:
https://jaeger.dev.storj.io/trace/73dd0c7413e4b545
FAIL (2026-01-06 16:37:49 UTC)
orbiter warning: context deadline exceeded
Duration: 11.595126802s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX.XX.XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXX] duration:
5.001025039s trace:
https://jaeger.dev.storj.io/trace/4e952f4f3b7b3068
FAIL (2026-01-06 16:37:41 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001049376s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.X.....X......................X......X......................] duration:
29.312720022s trace:
https://jaeger.dev.storj.io/trace/d5e2d8b8127d900
FAIL (2026-01-06 16:37:25 UTC)
orbiter warning: context deadline exceeded
Duration: 29.312748066s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
11 out of 60 is failed (threshold=3) [.......X......X..X.X.X........X.....XX......X..........XX...] duration:
48.314451697s trace:
https://jaeger.dev.storj.io/trace/7a3c6d288b21032f
FAIL (2026-01-06 16:37:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1z6uPcGsR78PiEpT46NBJEdyH33Re1YU5CtrMMFVeFf1KD3egC, Piece ID: SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA) context deadline exceeded; piecestore: (Node ID: 12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, Piece ID: 7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ) context deadline exceeded; piecestore: (Node ID: 12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, Piece ID: 6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA) context deadline exceeded; piecestore: (Node ID: 19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2, Piece ID: 75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA) context deadline exceeded
======LOGS======
N4JDOQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.885035615 (769h5m7.75611872s) [noise=true , address=193.149.164.143:4704 , piece_id=AR4IOO7WZ6JW7SPPD3MJNPJTOQN2QZKYRKPJZVABKTQIPDW3OEQQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.885639495 (769h5m7.75672253s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.886433417 (769h5m7.757516462s) [noise=true , address=145.79.227.40:4076 , piece_id=75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.886717688 (769h5m7.757800733s) [noise=true , address=77.90.31.19:4851 , piece_id=OIHGYRRDXJ4Q74XKYQR35CSMY26ZKZP3BDABQH46PAX7JE63GKTA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.887491352 (769h5m7.758574387s) [noise=true , address=145.79.227.9:4371 , piece_id=MCKXM54PMRLUKOKG57USLL3J2MRQLQZRDLU2HCETCPEKE2VFGD4Q, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.888279943 (769h5m7.759362988s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.889229281 (769h5m7.760312366s) [noise=true , address=145.79.227.10:4431 , piece_id=STAF5533UZXZ2KQM7LVMLPKOUJTS6XDZ6LLGIRNMJCRTJIJPIUBA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.889970773 (769h5m7.761053818s) [noise=true , address=77.90.31.13:4652 , piece_id=SYQETEG6NSQVRBIDOVSEIJJLI34NU7GG7ADOTG3NZI7MINIJFWJA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.890575594 (769h5m7.761658639s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891005143 (769h5m7.762088188s) [noise=true , address=145.79.227.42:4132 , piece_id=7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891231283 (769h5m7.762314328s) [noise=true , address=77.90.31.12:4612 , piece_id=FE5I4T5TOUK6E5H4YSL6BM2S3HPT4DV73APFNL42IGXS5PXXF3PQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891456172 (769h5m7.762539217s) [noise=true , address=145.79.227.44:4215 , piece_id=QHBK6LHYB7TG2BCU3WBN4DGBLVSIRGZFGX736KUNL3RFSQNPKWCQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891720314 (769h5m7.762803359s) [noise=true , address=145.79.227.51:4424 , piece_id=6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891740082 (769h5m7.762823117s) [noise=true , address=145.79.227.43:4162 , piece_id=VUK6VCFF47J4PPMA2OXWCLKGLK3CYNRRW6M5OX4MCHXT6JZCDDCQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892196090 (769h5m7.763279135s) [noise=true , address=145.79.227.32:4864 , piece_id=SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892243831 (769h5m7.763326916s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892724506 (769h5m7.763807541s) [noise=true , address=64.49.13.27:4965 , piece_id=LXGCIOG3WP5YCOXMPYGUJPSTNUELNABBAN6WWLFTC7G4NMARS3VA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:09.311257742 (769h5m9.182340777s) [upload=672885]: Next returned nil segment
Jan 6 16:34:09.311272190 (769h5m9.182355225s) [upload=672885]: Waiting for error group managing segments...
Jan 6 16:34:09.669387707 (769h5m9.540470742s) [noise=true , address=64.49.13.27:4965 , piece_id=LXGCIOG3WP5YCOXMPYGUJPSTNUELNABBAN6WWLFTC7G4NMARS3VA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:09.959387050 (769h5m9.830470115s) [noise=true , address=145.79.227.18:4034 , piece_id=KQAPYDVPBHKER2SRSQEZHCBOLJROINC4SOIXBXF5GO6QRWN4JDOQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:09.964887985 (769h5m9.83597102s) [noise=true , address=145.79.227.10:4431 , piece_id=STAF5533UZXZ2KQM7LVMLPKOUJTS6XDZ6LLGIRNMJCRTJIJPIUBA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.052844868 (769h5m9.923927903s) [noise=true , address=77.90.31.12:4612 , piece_id=FE5I4T5TOUK6E5H4YSL6BM2S3HPT4DV73APFNL42IGXS5PXXF3PQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.117702277 (769h5m9.988785322s) [noise=true , address=77.90.31.19:4851 , piece_id=OIHGYRRDXJ4Q74XKYQR35CSMY26ZKZP3BDABQH46PAX7JE63GKTA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.274326112 (769h5m10.145409147s) [noise=true , address=145.79.227.42:4132 , piece_id=7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.500870347 (769h5m10.371953382s) [noise=true , address=77.90.31.13:4652 , piece_id=SYQETEG6NSQVRBIDOVSEIJJLI34NU7GG7ADOTG3NZI7MINIJFWJA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.632035827 (769h5m10.503118862s) [noise=true , address=193.149.164.143:4704 , piece_id=AR4IOO7WZ6JW7SPPD3MJNPJTOQN2QZKYRKPJZVABKTQIPDW3OEQQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.844343924 (769h5m10.715426959s) [noise=true , address=145.79.227.44:4215 , piece_id=QHBK6LHYB7TG2BCU3WBN4DGBLVSIRGZFGX736KUNL3RFSQNPKWCQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.995733642 (769h5m10.866816687s) [noise=true , address=145.79.227.32:4864 , piece_id=SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.386027101 (769h5m11.257110157s) [noise=true , address=145.79.227.51:4424 , piece_id=6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.591729941 (769h5m11.462812986s) [noise=true , address=145.79.227.40:4076 , piece_id=75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.648804660 (769h5m11.519887705s) [noise=true , address=145.79.227.43:4162 , piece_id=VUK6VCFF47J4PPMA2OXWCLKGLK3CYNRRW6M5OX4MCHXT6JZCDDCQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:12.194505799 (769h5m12.065588834s) [noise=true , address=145.79.227.9:4371 , piece_id=MCKXM54PMRLUKOKG57USLL3J2MRQLQZRDLU2HCETCPEKE2VFGD4Q, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:12.194526478 (769h5m12.065609513s) [seg_pos={0 0}, upload=672885]: Segment reached optimal threshold of 14 pieces.
Jan 6 16:34:12.194744533 (769h5m12.065827568s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aKqg1aNsv8Yi7fE4asGz8bijcPewWDvUukKcf5SFnzgARju43): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.194751486 (769h5m12.065834531s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.194795740 (769h5m12.065878785s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.194804257 (769h5m12.065887302s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195012674 (769h5m12.066095719s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.195021781 (769h5m12.066104846s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195236690 (769h5m12.066319725s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.195244916 (769h5m12.066327951s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195274552 (769h5m12.066357597s) [seg_pos={0 0}, upload=672885]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 16:34:12.195381716 (769h5m12.066464761s) [upload=672885]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:12.195389681 (769h5m12.066472716s) [upload=672885]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:12.484471858 (769h5m12.355554903s) [upload=672885]: Done uploading segments...
Jan 6 16:34:12.484491015 (769h5m12.35557405s) [upload=672885]: Upload finished. err: <nil>
Duration: 48.314467437s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [...X...X..X.XXXX..X.....XXX........X.......X..XX.....XX.X...] duration:
16.07606531s trace:
https://jaeger.dev.storj.io/trace/36dd1cd45f4420cc
FAIL (2026-01-06 16:37:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ) context deadline exceeded; piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ) context deadline exceeded; piecestore: (Node ID: 1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, Piece ID: VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA) context deadline exceeded; piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ) context deadline exceeded
======LOGS======
Jan 6 16:34:23.725664194 (769h5m23.596747239s) [upload=672890]: Starting upload
Jan 6 16:34:23.725751641 (769h5m23.596834686s) [upload=672890]: Done starting upload
Jan 6 16:34:23.725958084 (769h5m23.59704112s) [upload=672890]: Uploading segments...
Jan 6 16:34:23.726102068 (769h5m23.597185104s) [upload=672890]: Got next segment. Inline: false
Jan 6 16:34:23.726139179 (769h5m23.597222214s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 16:34:23.726144118 (769h5m23.597227163s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 16:34:23.965881595 (769h5m23.83696463s) [seg_pos={0 0}, upload=672890]: Begin upload segment...
Jan 6 16:34:23.965936098 (769h5m23.837019143s) [seg_pos={0 0}, upload=672890]: Done begin upload segment.
Jan 6 16:34:23.965966356 (769h5m23.837049391s) [noise=true , address=145.79.227.6:4257 , piece_id=G3G5PW7TTBUUX43WRS27Q7IRERII2CSPOHSJF7CCW2JDS4SF76XA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.966002936 (769h5m23.837085971s) [noise=true , address=45.91.116.9:4262 , piece_id=2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.966726544 (769h5m23.837809579s) [noise=true , address=145.79.227.50:4375 , piece_id=UDJJ32G2LQWIENRTKANFOOXUGQHMGFFGNCVSIWHS74J72JVBL3EA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967009302 (769h5m23.838092347s) [noise=true , address=145.79.227.19:4079 , piece_id=RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967390548 (769h5m23.838473583s) [noise=true , address=145.79.227.28:4725 , piece_id=GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967734463 (769h5m23.838817498s) [noise=true , address=145.79.227.41:4086 , piece_id=VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967991642 (769h5m23.839074677s) [noise=true , address=145.79.227.52:4447 , piece_id=FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968251918 (769h5m23.839334953s) [noise=true , address=145.79.227.13:4802 , piece_id=BIDGVMI2OHM6ATPAXETBJE4SBPWVXFA22YW3WEF7FI737JQEHOAA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968517413 (769h5m23.839600448s) [noise=true , address=145.79.227.14:4837 , piece_id=OA5CMTGROWVHFX4OK2IF5OIY2VQBA6WKUVEF523JDPMP3TKLRR7A, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968781998 (769h5m23.839865033s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969032384 (769h5m23.840115419s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969281709 (769h5m23.840364754s) [noise=true , address=45.91.116.4:4939 , piece_id=WWM7Z4XQOOKV4AVX4WPGWDORYN6T4YQERLR2UHPHOR62JBJWP4NA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969552525 (769h5m23.84063556s) [noise=true , address=145.79.227.46:4601 , piece_id=VV7VWRLJ37DHIR7LAKUL3B5DMMYICZWYWLV6JZUDZUUXCTBROIUA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969879137 (769h5m23.840962172s) [noise=true , address=145.79.227.25:4619 , piece_id=UHWNEWE6INQD7K4E75QFCWONUBHCVUH2HT2HOYROMBXY3YSXY5LA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.970130446 (769h5m23.841213481s) [noise=true , address=145.79.227.15:4876 , piece_id=2GEIIGCRC4KERAZOXNPCWUSYA2TZZQBU4SZ3SVK55OZJ6BIIUGWQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.970262667 (769h5m23.841345703s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:25.774712127 (769h5m25.645795162s) [upload=672890]: Next returned nil segment
Jan 6 16:34:25.774729981 (769h5m25.645813046s) [upload=672890]: Waiting for error group managing segments...
Jan 6 16:34:26.137767310 (769h5m26.008850345s) [noise=true , address=145.79.227.52:4447 , piece_id=FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.162432296 (769h5m26.033515612s) [noise=true , address=145.79.227.28:4725 , piece_id=GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.163113923 (769h5m26.034196958s) [noise=true , address=145.79.227.25:4619 , piece_id=UHWNEWE6INQD7K4E75QFCWONUBHCVUH2HT2HOYROMBXY3YSXY5LA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.193456581 (769h5m26.064539616s) [noise=true , address=145.79.227.19:4079 , piece_id=RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.199003423 (769h5m26.070086468s) [noise=true , address=145.79.227.41:4086 , piece_id=VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.287764277 (769h5m26.158847343s) [noise=true , address=145.79.227.50:4375 , piece_id=UDJJ32G2LQWIENRTKANFOOXUGQHMGFFGNCVSIWHS74J72JVBL3EA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.327125958 (769h5m26.198208993s) [noise=true , address=145.79.227.14:4837 , piece_id=OA5CMTGROWVHFX4OK2IF5OIY2VQBA6WKUVEF523JDPMP3TKLRR7A, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.438912407 (769h5m26.309995442s) [noise=true , address=145.79.227.46:4601 , piece_id=VV7VWRLJ37DHIR7LAKUL3B5DMMYICZWYWLV6JZUDZUUXCTBROIUA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.496250988 (769h5m26.367334023s) [noise=true , address=45.91.116.4:4939 , piece_id=WWM7Z4XQOOKV4AVX4WPGWDORYN6T4YQERLR2UHPHOR62JBJWP4NA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.831068722 (769h5m26.702151768s) [noise=true , address=45.91.116.9:4262 , piece_id=2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.996782847 (769h5m26.867866524s) [noise=true , address=145.79.227.13:4802 , piece_id=BIDGVMI2OHM6ATPAXETBJE4SBPWVXFA22YW3WEF7FI737JQEHOAA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.030623856 (769h5m26.901706911s) [noise=true , address=145.79.227.6:4257 , piece_id=G3G5PW7TTBUUX43WRS27Q7IRERII2CSPOHSJF7CCW2JDS4SF76XA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.219843789 (769h5m27.090926824s) [noise=true , address=145.79.227.15:4876 , piece_id=2GEIIGCRC4KERAZOXNPCWUSYA2TZZQBU4SZ3SVK55OZJ6BIIUGWQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.219859249 (769h5m27.090942284s) [seg_pos={0 0}, upload=672890]: Segment reached optimal threshold of 13 pieces.
Jan 6 16:34:27.219990048 (769h5m27.091073093s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:27.219995408 (769h5m27.091078454s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220223452 (769h5m27.091306487s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:27.220229183 (769h5m27.091312228s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220717242 (769h5m27.091800277s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: write tcp 188.166.225.170:59640->145.79.227.47:4265: use of closed network connection; write tcp 188.166.225.170:59640->145.79.227.47:4265: use of closed network connection
Jan 6 16:34:27.220736199 (769h5m27.091819244s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220799980 (769h5m27.091883015s) [seg_pos={0 0}, upload=672890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 16:34:27.220910681 (769h5m27.091993716s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:27.220918155 (769h5m27.0920012s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:27.503741505 (769h5m27.37482454s) [upload=672890]: Done uploading segments...
Jan 6 16:34:27.503759960 (769h5m27.374842995s) [upload=672890]: Upload finished. err: <nil>
Duration: 16.07611308s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.....XX......................X.X....X.X.X....X.X...X....XXX.] duration:
12.37805507s trace:
https://jaeger.dev.storj.io/trace/73dd0c7413e4b545
FAIL (2026-01-06 16:37:15 UTC)
orbiter warning: context deadline exceeded
Duration: 12.378088503s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX.XX.XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration:
5.000647201s trace:
https://jaeger.dev.storj.io/trace/23bd45d94557d4e0
FAIL (2026-01-06 16:37:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000680705s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
12.772781739s trace:
https://jaeger.dev.storj.io/trace/4b3b841c440bfc62
FAIL (2026-01-06 16:36:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913312874 (769h5m40.784395909s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913698277 (769h5m40.784781322s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913862180 (769h5m40.784945215s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914346662 (769h5m40.785429707s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914842456 (769h5m40.785925491s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915307461 (769h5m40.786390506s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915793817 (769h5m40.786876863s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916279973 (769h5m40.787363018s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916759336 (769h5m40.787842371s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.920089677 (769h5m40.791172722s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:41.741185400 (769h5m41.612268445s) [upload=672894]: Next returned nil segment
Jan 6 16:34:41.741201020 (769h5m41.612284055s) [upload=672894]: Waiting for error group managing segments...
Jan 6 16:34:42.136024704 (769h5m42.00710774s) [noise=true , address=185.167.204.205:4036 , piece_id=4XKAIZPEJZAVAS7P2QILKY7XIDIIRIRWH7HTN5OVXUHAL2Y3H4SQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137413950 (769h5m42.008496995s) [noise=true , address=185.167.204.206:4051 , piece_id=XWKJM6ZIHISLSESJ2ZQQQMZTGMK5MHH6L4ZD5RXJ3SMAVASVVVOA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137527506 (769h5m42.008610541s) [noise=true , address=185.167.204.206:4049 , piece_id=4T665GS7XTBVEHUVT6SFIQW4F3TTBPDKZ57PKQHHHWWKC4P4JQPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.138183485 (769h5m42.00926653s) [noise=true , address=185.167.204.206:4048 , piece_id=ZGQYUYRK5PK24UF4PYT67FHNPR4GVFDCJ2OEGRFENLB2GYR2LFVQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.139902609 (769h5m42.010985644s) [noise=true , address=185.167.204.206:4056 , piece_id=KYHDW5RLJTZCURXCFIP7HN3ZYLPPHK7FTO6IIGQ5VJJUPF4ICQXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.140277843 (769h5m42.011360878s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.142394303 (769h5m42.013477348s) [noise=true , address=185.167.204.205:4021 , piece_id=U4X3NZRT7MHG2JTNRVDRYIC72UDMHUQUCY4VFW7JFVLLVACZN7OA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.179529514 (769h5m42.050612549s) [noise=true , address=185.167.204.206:4053 , piece_id=HBFCBJCOLPJLNE4INHP4JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.194387988 (769h5m42.065471023s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.197860059 (769h5m42.068943094s) [noise=true , address=185.167.204.206:4064 , piece_id=JGCXE3QUORVKLRIWEXFZUCT4XUF3DRVA35ZKQU4DQODLZBOH5I4A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.827615690 (769h5m42.698698725s) [noise=true , address=81.171.14.70:4041 , piece_id=NLGMNQJDFISSAHBX3EC6WMWJLP74FMR4MRPUIX4GCCCQRG5RRRCQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.917676689 (769h5m42.788759725s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.932213350 (769h5m42.803296385s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.956051121 (769h5m42.827134196s) [noise=true , address=81.171.14.70:4031 , piece_id=OJKJHNVWTPDITZC5CTQNB7UFPIVZDPE54ZXS4LWRM3XHYAB6443A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.005259852 (769h5m42.876342887s) [noise=true , address=185.167.204.205:4019 , piece_id=I7OP7AQS2DC3C7GZWG5RVJSLDKAZD2434WMBPRYMZFVAJYJG3JHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.052708792 (769h5m42.923791837s) [noise=true , address=81.171.14.70:4046 , piece_id=OV3PMCD6UIFNSQNTCOS3GEQC4676KFJOUH5ZX576OFOYC5EXCZPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.639749858 (769h5m43.510832903s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.719270598 (769h5m43.590353643s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.892052420 (769h5m43.763135466s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.901787498 (769h5m43.772870533s) [noise=true , address=81.171.14.70:4039 , piece_id=VFXOSS5DRL43RDJGPLSH3667HWKAF5ZNC2CF5BA7TDR4NSCG75HA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.019887208 (769h5m43.890970243s) [noise=true , address=185.167.204.206:4050 , piece_id=KSCVNGXUGFV7XRNU5MFIKLLQGZ5BNHTIXQLVHWPMITZOQUA23DGQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.510337921 (769h5m44.381420957s) [noise=true , address=185.167.204.205:4028 , piece_id=QPB4EPPEENW277S4SBIOCWWMBLIE6GKCGEM3TV3MXN6SYFMQAI7Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.692198242 (769h5m44.563281277s) [noise=true , address=89.149.201.147:4010 , piece_id=C57ORVBIZE4QUWCVG3OVG3KW2Z2GYVWUXEOYG5R53KV2WUQLR7JQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.394574949 (769h5m46.265657984s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.543864889 (769h5m46.414947934s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846238946 (769h5m46.717321982s) [noise=true , address=185.167.204.205:4006 , piece_id=W3XXEI7MG37VJHHK2CECNKIIHB7WSQABGVB5W5ZGAWSTFQ4NQMEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846258273 (769h5m46.717341308s) [seg_pos={0 0}, upload=672894]: Segment reached optimal threshold of 26 pieces.
Jan 6 16:34:46.846448986 (769h5m46.717532021s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846454998 (769h5m46.717538033s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846579175 (769h5m46.71766221s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846595475 (769h5m46.71767851s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846620453 (769h5m46.717703488s) [seg_pos={0 0}, upload=672894]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Jan 6 16:34:46.846729380 (769h5m46.717812415s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:46.846734600 (769h5m46.717817635s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:47.079572108 (769h5m46.950655143s) [upload=672894]: Done uploading segments...
Jan 6 16:34:47.079590854 (769h5m46.950673889s) [upload=672894]: Upload finished. err: <nil>
Duration: 12.77280881s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX.XX.XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration:
5.00149289s trace:
https://jaeger.dev.storj.io/trace/103a8f5ed8fc4a9b
FAIL (2026-01-06 16:36:56 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001512507s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..X.....X......................X......X.....................] duration:
22.557399495s trace:
https://jaeger.dev.storj.io/trace/d5e2d8b8127d900
FAIL (2026-01-06 16:36:49 UTC)
orbiter warning: context deadline exceeded
Duration: 22.557432278s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [......XX......................X.X....X.X.X....X.X...X....XXX] duration:
30.001470201s trace:
https://jaeger.dev.storj.io/trace/73dd0c7413e4b545
FAIL (2026-01-06 16:36:43 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001489147s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
11 out of 60 is failed (threshold=3) [........X......X..X.X.X........X.....XX......X..........XX..] duration:
40.897950004s trace:
https://jaeger.dev.storj.io/trace/7a3c6d288b21032f
FAIL (2026-01-06 16:36:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1z6uPcGsR78PiEpT46NBJEdyH33Re1YU5CtrMMFVeFf1KD3egC, Piece ID: SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA) context deadline exceeded; piecestore: (Node ID: 12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, Piece ID: 7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ) context deadline exceeded; piecestore: (Node ID: 12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, Piece ID: 6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA) context deadline exceeded; piecestore: (Node ID: 19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2, Piece ID: 75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA) context deadline exceeded
======LOGS======
N4JDOQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.885035615 (769h5m7.75611872s) [noise=true , address=193.149.164.143:4704 , piece_id=AR4IOO7WZ6JW7SPPD3MJNPJTOQN2QZKYRKPJZVABKTQIPDW3OEQQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.885639495 (769h5m7.75672253s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.886433417 (769h5m7.757516462s) [noise=true , address=145.79.227.40:4076 , piece_id=75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.886717688 (769h5m7.757800733s) [noise=true , address=77.90.31.19:4851 , piece_id=OIHGYRRDXJ4Q74XKYQR35CSMY26ZKZP3BDABQH46PAX7JE63GKTA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.887491352 (769h5m7.758574387s) [noise=true , address=145.79.227.9:4371 , piece_id=MCKXM54PMRLUKOKG57USLL3J2MRQLQZRDLU2HCETCPEKE2VFGD4Q, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.888279943 (769h5m7.759362988s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.889229281 (769h5m7.760312366s) [noise=true , address=145.79.227.10:4431 , piece_id=STAF5533UZXZ2KQM7LVMLPKOUJTS6XDZ6LLGIRNMJCRTJIJPIUBA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.889970773 (769h5m7.761053818s) [noise=true , address=77.90.31.13:4652 , piece_id=SYQETEG6NSQVRBIDOVSEIJJLI34NU7GG7ADOTG3NZI7MINIJFWJA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.890575594 (769h5m7.761658639s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891005143 (769h5m7.762088188s) [noise=true , address=145.79.227.42:4132 , piece_id=7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891231283 (769h5m7.762314328s) [noise=true , address=77.90.31.12:4612 , piece_id=FE5I4T5TOUK6E5H4YSL6BM2S3HPT4DV73APFNL42IGXS5PXXF3PQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891456172 (769h5m7.762539217s) [noise=true , address=145.79.227.44:4215 , piece_id=QHBK6LHYB7TG2BCU3WBN4DGBLVSIRGZFGX736KUNL3RFSQNPKWCQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891720314 (769h5m7.762803359s) [noise=true , address=145.79.227.51:4424 , piece_id=6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891740082 (769h5m7.762823117s) [noise=true , address=145.79.227.43:4162 , piece_id=VUK6VCFF47J4PPMA2OXWCLKGLK3CYNRRW6M5OX4MCHXT6JZCDDCQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892196090 (769h5m7.763279135s) [noise=true , address=145.79.227.32:4864 , piece_id=SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892243831 (769h5m7.763326916s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892724506 (769h5m7.763807541s) [noise=true , address=64.49.13.27:4965 , piece_id=LXGCIOG3WP5YCOXMPYGUJPSTNUELNABBAN6WWLFTC7G4NMARS3VA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:09.311257742 (769h5m9.182340777s) [upload=672885]: Next returned nil segment
Jan 6 16:34:09.311272190 (769h5m9.182355225s) [upload=672885]: Waiting for error group managing segments...
Jan 6 16:34:09.669387707 (769h5m9.540470742s) [noise=true , address=64.49.13.27:4965 , piece_id=LXGCIOG3WP5YCOXMPYGUJPSTNUELNABBAN6WWLFTC7G4NMARS3VA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:09.959387050 (769h5m9.830470115s) [noise=true , address=145.79.227.18:4034 , piece_id=KQAPYDVPBHKER2SRSQEZHCBOLJROINC4SOIXBXF5GO6QRWN4JDOQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:09.964887985 (769h5m9.83597102s) [noise=true , address=145.79.227.10:4431 , piece_id=STAF5533UZXZ2KQM7LVMLPKOUJTS6XDZ6LLGIRNMJCRTJIJPIUBA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.052844868 (769h5m9.923927903s) [noise=true , address=77.90.31.12:4612 , piece_id=FE5I4T5TOUK6E5H4YSL6BM2S3HPT4DV73APFNL42IGXS5PXXF3PQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.117702277 (769h5m9.988785322s) [noise=true , address=77.90.31.19:4851 , piece_id=OIHGYRRDXJ4Q74XKYQR35CSMY26ZKZP3BDABQH46PAX7JE63GKTA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.274326112 (769h5m10.145409147s) [noise=true , address=145.79.227.42:4132 , piece_id=7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.500870347 (769h5m10.371953382s) [noise=true , address=77.90.31.13:4652 , piece_id=SYQETEG6NSQVRBIDOVSEIJJLI34NU7GG7ADOTG3NZI7MINIJFWJA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.632035827 (769h5m10.503118862s) [noise=true , address=193.149.164.143:4704 , piece_id=AR4IOO7WZ6JW7SPPD3MJNPJTOQN2QZKYRKPJZVABKTQIPDW3OEQQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.844343924 (769h5m10.715426959s) [noise=true , address=145.79.227.44:4215 , piece_id=QHBK6LHYB7TG2BCU3WBN4DGBLVSIRGZFGX736KUNL3RFSQNPKWCQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.995733642 (769h5m10.866816687s) [noise=true , address=145.79.227.32:4864 , piece_id=SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.386027101 (769h5m11.257110157s) [noise=true , address=145.79.227.51:4424 , piece_id=6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.591729941 (769h5m11.462812986s) [noise=true , address=145.79.227.40:4076 , piece_id=75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.648804660 (769h5m11.519887705s) [noise=true , address=145.79.227.43:4162 , piece_id=VUK6VCFF47J4PPMA2OXWCLKGLK3CYNRRW6M5OX4MCHXT6JZCDDCQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:12.194505799 (769h5m12.065588834s) [noise=true , address=145.79.227.9:4371 , piece_id=MCKXM54PMRLUKOKG57USLL3J2MRQLQZRDLU2HCETCPEKE2VFGD4Q, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:12.194526478 (769h5m12.065609513s) [seg_pos={0 0}, upload=672885]: Segment reached optimal threshold of 14 pieces.
Jan 6 16:34:12.194744533 (769h5m12.065827568s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aKqg1aNsv8Yi7fE4asGz8bijcPewWDvUukKcf5SFnzgARju43): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.194751486 (769h5m12.065834531s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.194795740 (769h5m12.065878785s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.194804257 (769h5m12.065887302s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195012674 (769h5m12.066095719s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.195021781 (769h5m12.066104846s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195236690 (769h5m12.066319725s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.195244916 (769h5m12.066327951s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195274552 (769h5m12.066357597s) [seg_pos={0 0}, upload=672885]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 16:34:12.195381716 (769h5m12.066464761s) [upload=672885]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:12.195389681 (769h5m12.066472716s) [upload=672885]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:12.484471858 (769h5m12.355554903s) [upload=672885]: Done uploading segments...
Jan 6 16:34:12.484491015 (769h5m12.35557405s) [upload=672885]: Upload finished. err: <nil>
Duration: 40.897967879s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.......XX......................X.X....X.X.X....X.X...X....XX] duration:
30.001300404s trace:
https://jaeger.dev.storj.io/trace/591687990ae8792d
FAIL (2026-01-06 16:36:09 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001323829s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX.XX.XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration:
5.001291706s trace:
https://jaeger.dev.storj.io/trace/38387120eceb325e
FAIL (2026-01-06 16:36:03 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001312034s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...X.....X......................X......X....................] duration:
17.174627793s trace:
https://jaeger.dev.storj.io/trace/d5e2d8b8127d900
FAIL (2026-01-06 16:36:03 UTC)
orbiter warning: context deadline exceeded
Duration: 17.17465316s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X...X...X..X.XXXX..X.....XXX........X.......X..XX.....XX.X..] duration:
6.082731186s trace:
https://jaeger.dev.storj.io/trace/36dd1cd45f4420cc
FAIL (2026-01-06 16:35:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ) context deadline exceeded; piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ) context deadline exceeded; piecestore: (Node ID: 1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, Piece ID: VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA) context deadline exceeded; piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ) context deadline exceeded
======LOGS======
Jan 6 16:34:23.725664194 (769h5m23.596747239s) [upload=672890]: Starting upload
Jan 6 16:34:23.725751641 (769h5m23.596834686s) [upload=672890]: Done starting upload
Jan 6 16:34:23.725958084 (769h5m23.59704112s) [upload=672890]: Uploading segments...
Jan 6 16:34:23.726102068 (769h5m23.597185104s) [upload=672890]: Got next segment. Inline: false
Jan 6 16:34:23.726139179 (769h5m23.597222214s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 16:34:23.726144118 (769h5m23.597227163s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 16:34:23.965881595 (769h5m23.83696463s) [seg_pos={0 0}, upload=672890]: Begin upload segment...
Jan 6 16:34:23.965936098 (769h5m23.837019143s) [seg_pos={0 0}, upload=672890]: Done begin upload segment.
Jan 6 16:34:23.965966356 (769h5m23.837049391s) [noise=true , address=145.79.227.6:4257 , piece_id=G3G5PW7TTBUUX43WRS27Q7IRERII2CSPOHSJF7CCW2JDS4SF76XA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.966002936 (769h5m23.837085971s) [noise=true , address=45.91.116.9:4262 , piece_id=2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.966726544 (769h5m23.837809579s) [noise=true , address=145.79.227.50:4375 , piece_id=UDJJ32G2LQWIENRTKANFOOXUGQHMGFFGNCVSIWHS74J72JVBL3EA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967009302 (769h5m23.838092347s) [noise=true , address=145.79.227.19:4079 , piece_id=RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967390548 (769h5m23.838473583s) [noise=true , address=145.79.227.28:4725 , piece_id=GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967734463 (769h5m23.838817498s) [noise=true , address=145.79.227.41:4086 , piece_id=VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967991642 (769h5m23.839074677s) [noise=true , address=145.79.227.52:4447 , piece_id=FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968251918 (769h5m23.839334953s) [noise=true , address=145.79.227.13:4802 , piece_id=BIDGVMI2OHM6ATPAXETBJE4SBPWVXFA22YW3WEF7FI737JQEHOAA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968517413 (769h5m23.839600448s) [noise=true , address=145.79.227.14:4837 , piece_id=OA5CMTGROWVHFX4OK2IF5OIY2VQBA6WKUVEF523JDPMP3TKLRR7A, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968781998 (769h5m23.839865033s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969032384 (769h5m23.840115419s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969281709 (769h5m23.840364754s) [noise=true , address=45.91.116.4:4939 , piece_id=WWM7Z4XQOOKV4AVX4WPGWDORYN6T4YQERLR2UHPHOR62JBJWP4NA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969552525 (769h5m23.84063556s) [noise=true , address=145.79.227.46:4601 , piece_id=VV7VWRLJ37DHIR7LAKUL3B5DMMYICZWYWLV6JZUDZUUXCTBROIUA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969879137 (769h5m23.840962172s) [noise=true , address=145.79.227.25:4619 , piece_id=UHWNEWE6INQD7K4E75QFCWONUBHCVUH2HT2HOYROMBXY3YSXY5LA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.970130446 (769h5m23.841213481s) [noise=true , address=145.79.227.15:4876 , piece_id=2GEIIGCRC4KERAZOXNPCWUSYA2TZZQBU4SZ3SVK55OZJ6BIIUGWQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.970262667 (769h5m23.841345703s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:25.774712127 (769h5m25.645795162s) [upload=672890]: Next returned nil segment
Jan 6 16:34:25.774729981 (769h5m25.645813046s) [upload=672890]: Waiting for error group managing segments...
Jan 6 16:34:26.137767310 (769h5m26.008850345s) [noise=true , address=145.79.227.52:4447 , piece_id=FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.162432296 (769h5m26.033515612s) [noise=true , address=145.79.227.28:4725 , piece_id=GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.163113923 (769h5m26.034196958s) [noise=true , address=145.79.227.25:4619 , piece_id=UHWNEWE6INQD7K4E75QFCWONUBHCVUH2HT2HOYROMBXY3YSXY5LA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.193456581 (769h5m26.064539616s) [noise=true , address=145.79.227.19:4079 , piece_id=RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.199003423 (769h5m26.070086468s) [noise=true , address=145.79.227.41:4086 , piece_id=VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.287764277 (769h5m26.158847343s) [noise=true , address=145.79.227.50:4375 , piece_id=UDJJ32G2LQWIENRTKANFOOXUGQHMGFFGNCVSIWHS74J72JVBL3EA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.327125958 (769h5m26.198208993s) [noise=true , address=145.79.227.14:4837 , piece_id=OA5CMTGROWVHFX4OK2IF5OIY2VQBA6WKUVEF523JDPMP3TKLRR7A, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.438912407 (769h5m26.309995442s) [noise=true , address=145.79.227.46:4601 , piece_id=VV7VWRLJ37DHIR7LAKUL3B5DMMYICZWYWLV6JZUDZUUXCTBROIUA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.496250988 (769h5m26.367334023s) [noise=true , address=45.91.116.4:4939 , piece_id=WWM7Z4XQOOKV4AVX4WPGWDORYN6T4YQERLR2UHPHOR62JBJWP4NA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.831068722 (769h5m26.702151768s) [noise=true , address=45.91.116.9:4262 , piece_id=2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.996782847 (769h5m26.867866524s) [noise=true , address=145.79.227.13:4802 , piece_id=BIDGVMI2OHM6ATPAXETBJE4SBPWVXFA22YW3WEF7FI737JQEHOAA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.030623856 (769h5m26.901706911s) [noise=true , address=145.79.227.6:4257 , piece_id=G3G5PW7TTBUUX43WRS27Q7IRERII2CSPOHSJF7CCW2JDS4SF76XA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.219843789 (769h5m27.090926824s) [noise=true , address=145.79.227.15:4876 , piece_id=2GEIIGCRC4KERAZOXNPCWUSYA2TZZQBU4SZ3SVK55OZJ6BIIUGWQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.219859249 (769h5m27.090942284s) [seg_pos={0 0}, upload=672890]: Segment reached optimal threshold of 13 pieces.
Jan 6 16:34:27.219990048 (769h5m27.091073093s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:27.219995408 (769h5m27.091078454s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220223452 (769h5m27.091306487s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:27.220229183 (769h5m27.091312228s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220717242 (769h5m27.091800277s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: write tcp 188.166.225.170:59640->145.79.227.47:4265: use of closed network connection; write tcp 188.166.225.170:59640->145.79.227.47:4265: use of closed network connection
Jan 6 16:34:27.220736199 (769h5m27.091819244s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220799980 (769h5m27.091883015s) [seg_pos={0 0}, upload=672890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 16:34:27.220910681 (769h5m27.091993716s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:27.220918155 (769h5m27.0920012s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:27.503741505 (769h5m27.37482454s) [upload=672890]: Done uploading segments...
Jan 6 16:34:27.503759960 (769h5m27.374842995s) [upload=672890]: Upload finished. err: <nil>
Duration: 6.082757727s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
11 out of 60 is failed (threshold=3) [.........X......X..X.X.X........X.....XX......X..........XX.] duration:
47.550198815s trace:
https://jaeger.dev.storj.io/trace/7a3c6d288b21032f
FAIL (2026-01-06 16:35:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1z6uPcGsR78PiEpT46NBJEdyH33Re1YU5CtrMMFVeFf1KD3egC, Piece ID: SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA) context deadline exceeded; piecestore: (Node ID: 12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, Piece ID: 7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ) context deadline exceeded; piecestore: (Node ID: 12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, Piece ID: 6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA) context deadline exceeded; piecestore: (Node ID: 19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2, Piece ID: 75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA) context deadline exceeded
======LOGS======
N4JDOQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.885035615 (769h5m7.75611872s) [noise=true , address=193.149.164.143:4704 , piece_id=AR4IOO7WZ6JW7SPPD3MJNPJTOQN2QZKYRKPJZVABKTQIPDW3OEQQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.885639495 (769h5m7.75672253s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.886433417 (769h5m7.757516462s) [noise=true , address=145.79.227.40:4076 , piece_id=75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.886717688 (769h5m7.757800733s) [noise=true , address=77.90.31.19:4851 , piece_id=OIHGYRRDXJ4Q74XKYQR35CSMY26ZKZP3BDABQH46PAX7JE63GKTA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.887491352 (769h5m7.758574387s) [noise=true , address=145.79.227.9:4371 , piece_id=MCKXM54PMRLUKOKG57USLL3J2MRQLQZRDLU2HCETCPEKE2VFGD4Q, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.888279943 (769h5m7.759362988s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.889229281 (769h5m7.760312366s) [noise=true , address=145.79.227.10:4431 , piece_id=STAF5533UZXZ2KQM7LVMLPKOUJTS6XDZ6LLGIRNMJCRTJIJPIUBA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.889970773 (769h5m7.761053818s) [noise=true , address=77.90.31.13:4652 , piece_id=SYQETEG6NSQVRBIDOVSEIJJLI34NU7GG7ADOTG3NZI7MINIJFWJA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.890575594 (769h5m7.761658639s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891005143 (769h5m7.762088188s) [noise=true , address=145.79.227.42:4132 , piece_id=7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891231283 (769h5m7.762314328s) [noise=true , address=77.90.31.12:4612 , piece_id=FE5I4T5TOUK6E5H4YSL6BM2S3HPT4DV73APFNL42IGXS5PXXF3PQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891456172 (769h5m7.762539217s) [noise=true , address=145.79.227.44:4215 , piece_id=QHBK6LHYB7TG2BCU3WBN4DGBLVSIRGZFGX736KUNL3RFSQNPKWCQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891720314 (769h5m7.762803359s) [noise=true , address=145.79.227.51:4424 , piece_id=6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891740082 (769h5m7.762823117s) [noise=true , address=145.79.227.43:4162 , piece_id=VUK6VCFF47J4PPMA2OXWCLKGLK3CYNRRW6M5OX4MCHXT6JZCDDCQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892196090 (769h5m7.763279135s) [noise=true , address=145.79.227.32:4864 , piece_id=SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892243831 (769h5m7.763326916s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892724506 (769h5m7.763807541s) [noise=true , address=64.49.13.27:4965 , piece_id=LXGCIOG3WP5YCOXMPYGUJPSTNUELNABBAN6WWLFTC7G4NMARS3VA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:09.311257742 (769h5m9.182340777s) [upload=672885]: Next returned nil segment
Jan 6 16:34:09.311272190 (769h5m9.182355225s) [upload=672885]: Waiting for error group managing segments...
Jan 6 16:34:09.669387707 (769h5m9.540470742s) [noise=true , address=64.49.13.27:4965 , piece_id=LXGCIOG3WP5YCOXMPYGUJPSTNUELNABBAN6WWLFTC7G4NMARS3VA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:09.959387050 (769h5m9.830470115s) [noise=true , address=145.79.227.18:4034 , piece_id=KQAPYDVPBHKER2SRSQEZHCBOLJROINC4SOIXBXF5GO6QRWN4JDOQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:09.964887985 (769h5m9.83597102s) [noise=true , address=145.79.227.10:4431 , piece_id=STAF5533UZXZ2KQM7LVMLPKOUJTS6XDZ6LLGIRNMJCRTJIJPIUBA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.052844868 (769h5m9.923927903s) [noise=true , address=77.90.31.12:4612 , piece_id=FE5I4T5TOUK6E5H4YSL6BM2S3HPT4DV73APFNL42IGXS5PXXF3PQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.117702277 (769h5m9.988785322s) [noise=true , address=77.90.31.19:4851 , piece_id=OIHGYRRDXJ4Q74XKYQR35CSMY26ZKZP3BDABQH46PAX7JE63GKTA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.274326112 (769h5m10.145409147s) [noise=true , address=145.79.227.42:4132 , piece_id=7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.500870347 (769h5m10.371953382s) [noise=true , address=77.90.31.13:4652 , piece_id=SYQETEG6NSQVRBIDOVSEIJJLI34NU7GG7ADOTG3NZI7MINIJFWJA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.632035827 (769h5m10.503118862s) [noise=true , address=193.149.164.143:4704 , piece_id=AR4IOO7WZ6JW7SPPD3MJNPJTOQN2QZKYRKPJZVABKTQIPDW3OEQQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.844343924 (769h5m10.715426959s) [noise=true , address=145.79.227.44:4215 , piece_id=QHBK6LHYB7TG2BCU3WBN4DGBLVSIRGZFGX736KUNL3RFSQNPKWCQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.995733642 (769h5m10.866816687s) [noise=true , address=145.79.227.32:4864 , piece_id=SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.386027101 (769h5m11.257110157s) [noise=true , address=145.79.227.51:4424 , piece_id=6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.591729941 (769h5m11.462812986s) [noise=true , address=145.79.227.40:4076 , piece_id=75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.648804660 (769h5m11.519887705s) [noise=true , address=145.79.227.43:4162 , piece_id=VUK6VCFF47J4PPMA2OXWCLKGLK3CYNRRW6M5OX4MCHXT6JZCDDCQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:12.194505799 (769h5m12.065588834s) [noise=true , address=145.79.227.9:4371 , piece_id=MCKXM54PMRLUKOKG57USLL3J2MRQLQZRDLU2HCETCPEKE2VFGD4Q, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:12.194526478 (769h5m12.065609513s) [seg_pos={0 0}, upload=672885]: Segment reached optimal threshold of 14 pieces.
Jan 6 16:34:12.194744533 (769h5m12.065827568s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aKqg1aNsv8Yi7fE4asGz8bijcPewWDvUukKcf5SFnzgARju43): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.194751486 (769h5m12.065834531s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.194795740 (769h5m12.065878785s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.194804257 (769h5m12.065887302s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195012674 (769h5m12.066095719s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.195021781 (769h5m12.066104846s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195236690 (769h5m12.066319725s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.195244916 (769h5m12.066327951s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195274552 (769h5m12.066357597s) [seg_pos={0 0}, upload=672885]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 16:34:12.195381716 (769h5m12.066464761s) [upload=672885]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:12.195389681 (769h5m12.066472716s) [upload=672885]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:12.484471858 (769h5m12.355554903s) [upload=672885]: Done uploading segments...
Jan 6 16:34:12.484491015 (769h5m12.35557405s) [upload=672885]: Upload finished. err: <nil>
Duration: 47.550213955s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
15.517448731s trace:
https://jaeger.dev.storj.io/trace/4b3b841c440bfc62
FAIL (2026-01-06 16:35:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913312874 (769h5m40.784395909s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913698277 (769h5m40.784781322s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913862180 (769h5m40.784945215s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914346662 (769h5m40.785429707s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914842456 (769h5m40.785925491s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915307461 (769h5m40.786390506s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915793817 (769h5m40.786876863s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916279973 (769h5m40.787363018s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916759336 (769h5m40.787842371s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.920089677 (769h5m40.791172722s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:41.741185400 (769h5m41.612268445s) [upload=672894]: Next returned nil segment
Jan 6 16:34:41.741201020 (769h5m41.612284055s) [upload=672894]: Waiting for error group managing segments...
Jan 6 16:34:42.136024704 (769h5m42.00710774s) [noise=true , address=185.167.204.205:4036 , piece_id=4XKAIZPEJZAVAS7P2QILKY7XIDIIRIRWH7HTN5OVXUHAL2Y3H4SQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137413950 (769h5m42.008496995s) [noise=true , address=185.167.204.206:4051 , piece_id=XWKJM6ZIHISLSESJ2ZQQQMZTGMK5MHH6L4ZD5RXJ3SMAVASVVVOA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137527506 (769h5m42.008610541s) [noise=true , address=185.167.204.206:4049 , piece_id=4T665GS7XTBVEHUVT6SFIQW4F3TTBPDKZ57PKQHHHWWKC4P4JQPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.138183485 (769h5m42.00926653s) [noise=true , address=185.167.204.206:4048 , piece_id=ZGQYUYRK5PK24UF4PYT67FHNPR4GVFDCJ2OEGRFENLB2GYR2LFVQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.139902609 (769h5m42.010985644s) [noise=true , address=185.167.204.206:4056 , piece_id=KYHDW5RLJTZCURXCFIP7HN3ZYLPPHK7FTO6IIGQ5VJJUPF4ICQXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.140277843 (769h5m42.011360878s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.142394303 (769h5m42.013477348s) [noise=true , address=185.167.204.205:4021 , piece_id=U4X3NZRT7MHG2JTNRVDRYIC72UDMHUQUCY4VFW7JFVLLVACZN7OA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.179529514 (769h5m42.050612549s) [noise=true , address=185.167.204.206:4053 , piece_id=HBFCBJCOLPJLNE4INHP4JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.194387988 (769h5m42.065471023s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.197860059 (769h5m42.068943094s) [noise=true , address=185.167.204.206:4064 , piece_id=JGCXE3QUORVKLRIWEXFZUCT4XUF3DRVA35ZKQU4DQODLZBOH5I4A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.827615690 (769h5m42.698698725s) [noise=true , address=81.171.14.70:4041 , piece_id=NLGMNQJDFISSAHBX3EC6WMWJLP74FMR4MRPUIX4GCCCQRG5RRRCQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.917676689 (769h5m42.788759725s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.932213350 (769h5m42.803296385s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.956051121 (769h5m42.827134196s) [noise=true , address=81.171.14.70:4031 , piece_id=OJKJHNVWTPDITZC5CTQNB7UFPIVZDPE54ZXS4LWRM3XHYAB6443A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.005259852 (769h5m42.876342887s) [noise=true , address=185.167.204.205:4019 , piece_id=I7OP7AQS2DC3C7GZWG5RVJSLDKAZD2434WMBPRYMZFVAJYJG3JHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.052708792 (769h5m42.923791837s) [noise=true , address=81.171.14.70:4046 , piece_id=OV3PMCD6UIFNSQNTCOS3GEQC4676KFJOUH5ZX576OFOYC5EXCZPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.639749858 (769h5m43.510832903s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.719270598 (769h5m43.590353643s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.892052420 (769h5m43.763135466s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.901787498 (769h5m43.772870533s) [noise=true , address=81.171.14.70:4039 , piece_id=VFXOSS5DRL43RDJGPLSH3667HWKAF5ZNC2CF5BA7TDR4NSCG75HA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.019887208 (769h5m43.890970243s) [noise=true , address=185.167.204.206:4050 , piece_id=KSCVNGXUGFV7XRNU5MFIKLLQGZ5BNHTIXQLVHWPMITZOQUA23DGQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.510337921 (769h5m44.381420957s) [noise=true , address=185.167.204.205:4028 , piece_id=QPB4EPPEENW277S4SBIOCWWMBLIE6GKCGEM3TV3MXN6SYFMQAI7Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.692198242 (769h5m44.563281277s) [noise=true , address=89.149.201.147:4010 , piece_id=C57ORVBIZE4QUWCVG3OVG3KW2Z2GYVWUXEOYG5R53KV2WUQLR7JQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.394574949 (769h5m46.265657984s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.543864889 (769h5m46.414947934s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846238946 (769h5m46.717321982s) [noise=true , address=185.167.204.205:4006 , piece_id=W3XXEI7MG37VJHHK2CECNKIIHB7WSQABGVB5W5ZGAWSTFQ4NQMEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846258273 (769h5m46.717341308s) [seg_pos={0 0}, upload=672894]: Segment reached optimal threshold of 26 pieces.
Jan 6 16:34:46.846448986 (769h5m46.717532021s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846454998 (769h5m46.717538033s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846579175 (769h5m46.71766221s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846595475 (769h5m46.71767851s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846620453 (769h5m46.717703488s) [seg_pos={0 0}, upload=672894]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Jan 6 16:34:46.846729380 (769h5m46.717812415s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:46.846734600 (769h5m46.717817635s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:47.079572108 (769h5m46.950655143s) [upload=672894]: Done uploading segments...
Jan 6 16:34:47.079590854 (769h5m46.950673889s) [upload=672894]: Upload finished. err: <nil>
Duration: 15.517475993s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX.XX.XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXX] duration:
5.001269904s trace:
https://jaeger.dev.storj.io/trace/4d5bbc55aba5ebd9
FAIL (2026-01-06 16:35:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001291466s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
52.511088566s trace:
https://jaeger.dev.storj.io/trace/4b3b841c440bfc62
FAIL (2026-01-06 16:35:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913312874 (769h5m40.784395909s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913698277 (769h5m40.784781322s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.913862180 (769h5m40.784945215s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914346662 (769h5m40.785429707s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.914842456 (769h5m40.785925491s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915307461 (769h5m40.786390506s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.915793817 (769h5m40.786876863s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916279973 (769h5m40.787363018s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.916759336 (769h5m40.787842371s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:40.920089677 (769h5m40.791172722s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Uploading piece...
Jan 6 16:34:41.741185400 (769h5m41.612268445s) [upload=672894]: Next returned nil segment
Jan 6 16:34:41.741201020 (769h5m41.612284055s) [upload=672894]: Waiting for error group managing segments...
Jan 6 16:34:42.136024704 (769h5m42.00710774s) [noise=true , address=185.167.204.205:4036 , piece_id=4XKAIZPEJZAVAS7P2QILKY7XIDIIRIRWH7HTN5OVXUHAL2Y3H4SQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137413950 (769h5m42.008496995s) [noise=true , address=185.167.204.206:4051 , piece_id=XWKJM6ZIHISLSESJ2ZQQQMZTGMK5MHH6L4ZD5RXJ3SMAVASVVVOA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.137527506 (769h5m42.008610541s) [noise=true , address=185.167.204.206:4049 , piece_id=4T665GS7XTBVEHUVT6SFIQW4F3TTBPDKZ57PKQHHHWWKC4P4JQPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.138183485 (769h5m42.00926653s) [noise=true , address=185.167.204.206:4048 , piece_id=ZGQYUYRK5PK24UF4PYT67FHNPR4GVFDCJ2OEGRFENLB2GYR2LFVQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.139902609 (769h5m42.010985644s) [noise=true , address=185.167.204.206:4056 , piece_id=KYHDW5RLJTZCURXCFIP7HN3ZYLPPHK7FTO6IIGQ5VJJUPF4ICQXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.140277843 (769h5m42.011360878s) [noise=true , address=185.167.204.206:4067 , piece_id=GLITDKTEKYV4CFGL3TJXWJDFOBYPFX2UTNJFPUCZDJM3HOWVFS4Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.142394303 (769h5m42.013477348s) [noise=true , address=185.167.204.205:4021 , piece_id=U4X3NZRT7MHG2JTNRVDRYIC72UDMHUQUCY4VFW7JFVLLVACZN7OA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.179529514 (769h5m42.050612549s) [noise=true , address=185.167.204.206:4053 , piece_id=HBFCBJCOLPJLNE4INHP4JXEZNOOX75ENDV6EBXU4NR4HW4GTZMVA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.194387988 (769h5m42.065471023s) [noise=true , address=185.167.204.205:4035 , piece_id=HPHYG44QNKJYFSXEBH6FPYLFXLO6OUIVGDYL5GXJMP3PDNFUZEQA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.197860059 (769h5m42.068943094s) [noise=true , address=185.167.204.206:4064 , piece_id=JGCXE3QUORVKLRIWEXFZUCT4XUF3DRVA35ZKQU4DQODLZBOH5I4A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.827615690 (769h5m42.698698725s) [noise=true , address=81.171.14.70:4041 , piece_id=NLGMNQJDFISSAHBX3EC6WMWJLP74FMR4MRPUIX4GCCCQRG5RRRCQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.917676689 (769h5m42.788759725s) [noise=true , address=81.171.14.70:4034 , piece_id=5LGDXPCHMOQPZGJAEH3ZSPF64RSTVAJRQFZBXVWP5D3Y36YFLUUQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.932213350 (769h5m42.803296385s) [noise=true , address=81.171.14.70:4030 , piece_id=WV5BULPUQSAE7DUO2FWY74KF3MIHSQXYC673GNJ7XUBQXXM6KHEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:42.956051121 (769h5m42.827134196s) [noise=true , address=81.171.14.70:4031 , piece_id=OJKJHNVWTPDITZC5CTQNB7UFPIVZDPE54ZXS4LWRM3XHYAB6443A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.005259852 (769h5m42.876342887s) [noise=true , address=185.167.204.205:4019 , piece_id=I7OP7AQS2DC3C7GZWG5RVJSLDKAZD2434WMBPRYMZFVAJYJG3JHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.052708792 (769h5m42.923791837s) [noise=true , address=81.171.14.70:4046 , piece_id=OV3PMCD6UIFNSQNTCOS3GEQC4676KFJOUH5ZX576OFOYC5EXCZPQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.639749858 (769h5m43.510832903s) [noise=true , address=89.149.201.147:4004 , piece_id=NMPCYNSE3YVIAVDCWRO4VP6IN4DKZ2P4YIFYI6ZDTEU75QSFC4NQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.719270598 (769h5m43.590353643s) [noise=true , address=89.149.201.147:4019 , piece_id=LS3NXHELDI7H6VCTHO6LAGRDDG46ST4IF2XOFZDZ6EK756K5ACHQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.892052420 (769h5m43.763135466s) [noise=true , address=81.171.14.70:4035 , piece_id=BRXIJD7JX7L37MWMTVPL5KSESQUQEZNRXLBLL26XDMYIJSCRUD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:43.901787498 (769h5m43.772870533s) [noise=true , address=81.171.14.70:4039 , piece_id=VFXOSS5DRL43RDJGPLSH3667HWKAF5ZNC2CF5BA7TDR4NSCG75HA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.019887208 (769h5m43.890970243s) [noise=true , address=185.167.204.206:4050 , piece_id=KSCVNGXUGFV7XRNU5MFIKLLQGZ5BNHTIXQLVHWPMITZOQUA23DGQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.510337921 (769h5m44.381420957s) [noise=true , address=185.167.204.205:4028 , piece_id=QPB4EPPEENW277S4SBIOCWWMBLIE6GKCGEM3TV3MXN6SYFMQAI7Q, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:44.692198242 (769h5m44.563281277s) [noise=true , address=89.149.201.147:4010 , piece_id=C57ORVBIZE4QUWCVG3OVG3KW2Z2GYVWUXEOYG5R53KV2WUQLR7JQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.394574949 (769h5m46.265657984s) [noise=true , address=89.149.201.147:4014 , piece_id=KNV6OXYOC2H7LVZCW4ZWQRYRN5ZCMII3OO3FAS3YYQ22574KUCRQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.543864889 (769h5m46.414947934s) [noise=true , address=185.167.204.205:4014 , piece_id=XUCDQV4VW4ZZFTSZV2YEDAO5GQFP2ASW6BJFFC735UWEVIR3RNXA, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846238946 (769h5m46.717321982s) [noise=true , address=185.167.204.205:4006 , piece_id=W3XXEI7MG37VJHHK2CECNKIIHB7WSQABGVB5W5ZGAWSTFQ4NQMEQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: <nil>
Jan 6 16:34:46.846258273 (769h5m46.717341308s) [seg_pos={0 0}, upload=672894]: Segment reached optimal threshold of 26 pieces.
Jan 6 16:34:46.846448986 (769h5m46.717532021s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846454998 (769h5m46.717538033s) [noise=true , address=185.167.204.206:4072 , piece_id=EJ4XCLNDZETIFIKE3FK2Q4NA35KT5LOTNGE253MKUCAIGH4FZD5A, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846579175 (769h5m46.71766221s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:46.846595475 (769h5m46.71767851s) [noise=true , address=89.149.201.147:4005 , piece_id=HWCYTAY5UCUFZSKP423EG3HS7KKKFZZB3PDV5WDKJTQ3PYETMVIQ, seg_pos={0 0}, upload=672894]: Long tail cancellation due to optimal threshold
Jan 6 16:34:46.846620453 (769h5m46.717703488s) [seg_pos={0 0}, upload=672894]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Jan 6 16:34:46.846729380 (769h5m46.717812415s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:46.846734600 (769h5m46.717817635s) [upload=672894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:47.079572108 (769h5m46.950655143s) [upload=672894]: Done uploading segments...
Jan 6 16:34:47.079590854 (769h5m46.950673889s) [upload=672894]: Upload finished. err: <nil>
Duration: 52.511121328s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [X.......XX......................X.X....X.X.X....X.X...X....X] duration:
30.00138963s trace:
https://jaeger.dev.storj.io/trace/5a77d304bce1993b
FAIL (2026-01-06 16:35:32 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001402264s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [....X.....X......................X......X...................] duration:
17.747546822s trace:
https://jaeger.dev.storj.io/trace/d5e2d8b8127d900
FAIL (2026-01-06 16:35:27 UTC)
orbiter warning: context deadline exceeded
Duration: 17.747570988s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [XX...X...X..X.XXXX..X.....XXX........X.......X..XX.....XX.X.] duration:
6.21281166s trace:
https://jaeger.dev.storj.io/trace/36dd1cd45f4420cc
FAIL (2026-01-06 16:35:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ) context deadline exceeded; piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ) context deadline exceeded; piecestore: (Node ID: 1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, Piece ID: VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA) context deadline exceeded; piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ) context deadline exceeded
======LOGS======
Jan 6 16:34:23.725664194 (769h5m23.596747239s) [upload=672890]: Starting upload
Jan 6 16:34:23.725751641 (769h5m23.596834686s) [upload=672890]: Done starting upload
Jan 6 16:34:23.725958084 (769h5m23.59704112s) [upload=672890]: Uploading segments...
Jan 6 16:34:23.726102068 (769h5m23.597185104s) [upload=672890]: Got next segment. Inline: false
Jan 6 16:34:23.726139179 (769h5m23.597222214s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 16:34:23.726144118 (769h5m23.597227163s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 16:34:23.965881595 (769h5m23.83696463s) [seg_pos={0 0}, upload=672890]: Begin upload segment...
Jan 6 16:34:23.965936098 (769h5m23.837019143s) [seg_pos={0 0}, upload=672890]: Done begin upload segment.
Jan 6 16:34:23.965966356 (769h5m23.837049391s) [noise=true , address=145.79.227.6:4257 , piece_id=G3G5PW7TTBUUX43WRS27Q7IRERII2CSPOHSJF7CCW2JDS4SF76XA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.966002936 (769h5m23.837085971s) [noise=true , address=45.91.116.9:4262 , piece_id=2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.966726544 (769h5m23.837809579s) [noise=true , address=145.79.227.50:4375 , piece_id=UDJJ32G2LQWIENRTKANFOOXUGQHMGFFGNCVSIWHS74J72JVBL3EA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967009302 (769h5m23.838092347s) [noise=true , address=145.79.227.19:4079 , piece_id=RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967390548 (769h5m23.838473583s) [noise=true , address=145.79.227.28:4725 , piece_id=GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967734463 (769h5m23.838817498s) [noise=true , address=145.79.227.41:4086 , piece_id=VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967991642 (769h5m23.839074677s) [noise=true , address=145.79.227.52:4447 , piece_id=FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968251918 (769h5m23.839334953s) [noise=true , address=145.79.227.13:4802 , piece_id=BIDGVMI2OHM6ATPAXETBJE4SBPWVXFA22YW3WEF7FI737JQEHOAA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968517413 (769h5m23.839600448s) [noise=true , address=145.79.227.14:4837 , piece_id=OA5CMTGROWVHFX4OK2IF5OIY2VQBA6WKUVEF523JDPMP3TKLRR7A, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968781998 (769h5m23.839865033s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969032384 (769h5m23.840115419s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969281709 (769h5m23.840364754s) [noise=true , address=45.91.116.4:4939 , piece_id=WWM7Z4XQOOKV4AVX4WPGWDORYN6T4YQERLR2UHPHOR62JBJWP4NA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969552525 (769h5m23.84063556s) [noise=true , address=145.79.227.46:4601 , piece_id=VV7VWRLJ37DHIR7LAKUL3B5DMMYICZWYWLV6JZUDZUUXCTBROIUA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969879137 (769h5m23.840962172s) [noise=true , address=145.79.227.25:4619 , piece_id=UHWNEWE6INQD7K4E75QFCWONUBHCVUH2HT2HOYROMBXY3YSXY5LA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.970130446 (769h5m23.841213481s) [noise=true , address=145.79.227.15:4876 , piece_id=2GEIIGCRC4KERAZOXNPCWUSYA2TZZQBU4SZ3SVK55OZJ6BIIUGWQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.970262667 (769h5m23.841345703s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:25.774712127 (769h5m25.645795162s) [upload=672890]: Next returned nil segment
Jan 6 16:34:25.774729981 (769h5m25.645813046s) [upload=672890]: Waiting for error group managing segments...
Jan 6 16:34:26.137767310 (769h5m26.008850345s) [noise=true , address=145.79.227.52:4447 , piece_id=FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.162432296 (769h5m26.033515612s) [noise=true , address=145.79.227.28:4725 , piece_id=GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.163113923 (769h5m26.034196958s) [noise=true , address=145.79.227.25:4619 , piece_id=UHWNEWE6INQD7K4E75QFCWONUBHCVUH2HT2HOYROMBXY3YSXY5LA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.193456581 (769h5m26.064539616s) [noise=true , address=145.79.227.19:4079 , piece_id=RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.199003423 (769h5m26.070086468s) [noise=true , address=145.79.227.41:4086 , piece_id=VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.287764277 (769h5m26.158847343s) [noise=true , address=145.79.227.50:4375 , piece_id=UDJJ32G2LQWIENRTKANFOOXUGQHMGFFGNCVSIWHS74J72JVBL3EA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.327125958 (769h5m26.198208993s) [noise=true , address=145.79.227.14:4837 , piece_id=OA5CMTGROWVHFX4OK2IF5OIY2VQBA6WKUVEF523JDPMP3TKLRR7A, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.438912407 (769h5m26.309995442s) [noise=true , address=145.79.227.46:4601 , piece_id=VV7VWRLJ37DHIR7LAKUL3B5DMMYICZWYWLV6JZUDZUUXCTBROIUA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.496250988 (769h5m26.367334023s) [noise=true , address=45.91.116.4:4939 , piece_id=WWM7Z4XQOOKV4AVX4WPGWDORYN6T4YQERLR2UHPHOR62JBJWP4NA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.831068722 (769h5m26.702151768s) [noise=true , address=45.91.116.9:4262 , piece_id=2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.996782847 (769h5m26.867866524s) [noise=true , address=145.79.227.13:4802 , piece_id=BIDGVMI2OHM6ATPAXETBJE4SBPWVXFA22YW3WEF7FI737JQEHOAA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.030623856 (769h5m26.901706911s) [noise=true , address=145.79.227.6:4257 , piece_id=G3G5PW7TTBUUX43WRS27Q7IRERII2CSPOHSJF7CCW2JDS4SF76XA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.219843789 (769h5m27.090926824s) [noise=true , address=145.79.227.15:4876 , piece_id=2GEIIGCRC4KERAZOXNPCWUSYA2TZZQBU4SZ3SVK55OZJ6BIIUGWQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.219859249 (769h5m27.090942284s) [seg_pos={0 0}, upload=672890]: Segment reached optimal threshold of 13 pieces.
Jan 6 16:34:27.219990048 (769h5m27.091073093s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:27.219995408 (769h5m27.091078454s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220223452 (769h5m27.091306487s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:27.220229183 (769h5m27.091312228s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220717242 (769h5m27.091800277s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: write tcp 188.166.225.170:59640->145.79.227.47:4265: use of closed network connection; write tcp 188.166.225.170:59640->145.79.227.47:4265: use of closed network connection
Jan 6 16:34:27.220736199 (769h5m27.091819244s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220799980 (769h5m27.091883015s) [seg_pos={0 0}, upload=672890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 16:34:27.220910681 (769h5m27.091993716s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:27.220918155 (769h5m27.0920012s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:27.503741505 (769h5m27.37482454s) [upload=672890]: Done uploading segments...
Jan 6 16:34:27.503759960 (769h5m27.374842995s) [upload=672890]: Upload finished. err: <nil>
Duration: 6.212854462s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
11 out of 60 is failed (threshold=3) [..........X......X..X.X.X........X.....XX......X..........XX] duration:
1m0.002646212s trace:
https://jaeger.dev.storj.io/trace/7a3c6d288b21032f
FAIL (2026-01-06 16:35:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1z6uPcGsR78PiEpT46NBJEdyH33Re1YU5CtrMMFVeFf1KD3egC, Piece ID: SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA) context deadline exceeded; piecestore: (Node ID: 12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, Piece ID: 7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ) context deadline exceeded; piecestore: (Node ID: 12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, Piece ID: 6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA) context deadline exceeded; piecestore: (Node ID: 19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2, Piece ID: 75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA) context deadline exceeded
======LOGS======
N4JDOQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.885035615 (769h5m7.75611872s) [noise=true , address=193.149.164.143:4704 , piece_id=AR4IOO7WZ6JW7SPPD3MJNPJTOQN2QZKYRKPJZVABKTQIPDW3OEQQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.885639495 (769h5m7.75672253s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.886433417 (769h5m7.757516462s) [noise=true , address=145.79.227.40:4076 , piece_id=75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.886717688 (769h5m7.757800733s) [noise=true , address=77.90.31.19:4851 , piece_id=OIHGYRRDXJ4Q74XKYQR35CSMY26ZKZP3BDABQH46PAX7JE63GKTA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.887491352 (769h5m7.758574387s) [noise=true , address=145.79.227.9:4371 , piece_id=MCKXM54PMRLUKOKG57USLL3J2MRQLQZRDLU2HCETCPEKE2VFGD4Q, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.888279943 (769h5m7.759362988s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.889229281 (769h5m7.760312366s) [noise=true , address=145.79.227.10:4431 , piece_id=STAF5533UZXZ2KQM7LVMLPKOUJTS6XDZ6LLGIRNMJCRTJIJPIUBA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.889970773 (769h5m7.761053818s) [noise=true , address=77.90.31.13:4652 , piece_id=SYQETEG6NSQVRBIDOVSEIJJLI34NU7GG7ADOTG3NZI7MINIJFWJA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.890575594 (769h5m7.761658639s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891005143 (769h5m7.762088188s) [noise=true , address=145.79.227.42:4132 , piece_id=7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891231283 (769h5m7.762314328s) [noise=true , address=77.90.31.12:4612 , piece_id=FE5I4T5TOUK6E5H4YSL6BM2S3HPT4DV73APFNL42IGXS5PXXF3PQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891456172 (769h5m7.762539217s) [noise=true , address=145.79.227.44:4215 , piece_id=QHBK6LHYB7TG2BCU3WBN4DGBLVSIRGZFGX736KUNL3RFSQNPKWCQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891720314 (769h5m7.762803359s) [noise=true , address=145.79.227.51:4424 , piece_id=6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.891740082 (769h5m7.762823117s) [noise=true , address=145.79.227.43:4162 , piece_id=VUK6VCFF47J4PPMA2OXWCLKGLK3CYNRRW6M5OX4MCHXT6JZCDDCQ, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892196090 (769h5m7.763279135s) [noise=true , address=145.79.227.32:4864 , piece_id=SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892243831 (769h5m7.763326916s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:07.892724506 (769h5m7.763807541s) [noise=true , address=64.49.13.27:4965 , piece_id=LXGCIOG3WP5YCOXMPYGUJPSTNUELNABBAN6WWLFTC7G4NMARS3VA, seg_pos={0 0}, upload=672885]: Uploading piece...
Jan 6 16:34:09.311257742 (769h5m9.182340777s) [upload=672885]: Next returned nil segment
Jan 6 16:34:09.311272190 (769h5m9.182355225s) [upload=672885]: Waiting for error group managing segments...
Jan 6 16:34:09.669387707 (769h5m9.540470742s) [noise=true , address=64.49.13.27:4965 , piece_id=LXGCIOG3WP5YCOXMPYGUJPSTNUELNABBAN6WWLFTC7G4NMARS3VA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:09.959387050 (769h5m9.830470115s) [noise=true , address=145.79.227.18:4034 , piece_id=KQAPYDVPBHKER2SRSQEZHCBOLJROINC4SOIXBXF5GO6QRWN4JDOQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:09.964887985 (769h5m9.83597102s) [noise=true , address=145.79.227.10:4431 , piece_id=STAF5533UZXZ2KQM7LVMLPKOUJTS6XDZ6LLGIRNMJCRTJIJPIUBA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.052844868 (769h5m9.923927903s) [noise=true , address=77.90.31.12:4612 , piece_id=FE5I4T5TOUK6E5H4YSL6BM2S3HPT4DV73APFNL42IGXS5PXXF3PQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.117702277 (769h5m9.988785322s) [noise=true , address=77.90.31.19:4851 , piece_id=OIHGYRRDXJ4Q74XKYQR35CSMY26ZKZP3BDABQH46PAX7JE63GKTA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.274326112 (769h5m10.145409147s) [noise=true , address=145.79.227.42:4132 , piece_id=7WGCF343DAO6VW4HLSIG2ECBF32JOJSRLSAVB4HEOSKZPUOWGQRQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.500870347 (769h5m10.371953382s) [noise=true , address=77.90.31.13:4652 , piece_id=SYQETEG6NSQVRBIDOVSEIJJLI34NU7GG7ADOTG3NZI7MINIJFWJA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.632035827 (769h5m10.503118862s) [noise=true , address=193.149.164.143:4704 , piece_id=AR4IOO7WZ6JW7SPPD3MJNPJTOQN2QZKYRKPJZVABKTQIPDW3OEQQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.844343924 (769h5m10.715426959s) [noise=true , address=145.79.227.44:4215 , piece_id=QHBK6LHYB7TG2BCU3WBN4DGBLVSIRGZFGX736KUNL3RFSQNPKWCQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:10.995733642 (769h5m10.866816687s) [noise=true , address=145.79.227.32:4864 , piece_id=SEP53TXZ5V4QDSHNZCQ6APICGE4GESBL22QSBO5JZ5RJVMIR3ILA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.386027101 (769h5m11.257110157s) [noise=true , address=145.79.227.51:4424 , piece_id=6W64LEHLJXRRNSEFVHX2BNIX6HQMWO42YPOAHTK3CL76ASO42NQA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.591729941 (769h5m11.462812986s) [noise=true , address=145.79.227.40:4076 , piece_id=75G7G7TDVHWI2V3OM6OMVQJ365O25P47HLJK73D4GC4KSZ6OT2HA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:11.648804660 (769h5m11.519887705s) [noise=true , address=145.79.227.43:4162 , piece_id=VUK6VCFF47J4PPMA2OXWCLKGLK3CYNRRW6M5OX4MCHXT6JZCDDCQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:12.194505799 (769h5m12.065588834s) [noise=true , address=145.79.227.9:4371 , piece_id=MCKXM54PMRLUKOKG57USLL3J2MRQLQZRDLU2HCETCPEKE2VFGD4Q, seg_pos={0 0}, upload=672885]: Done uploading piece. err: <nil>
Jan 6 16:34:12.194526478 (769h5m12.065609513s) [seg_pos={0 0}, upload=672885]: Segment reached optimal threshold of 14 pieces.
Jan 6 16:34:12.194744533 (769h5m12.065827568s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aKqg1aNsv8Yi7fE4asGz8bijcPewWDvUukKcf5SFnzgARju43): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.194751486 (769h5m12.065834531s) [noise=true , address=145.79.227.30:4787 , piece_id=TM7WJURQCPCPFGTU46DJY272OOHOPDK25UZDVDIENKPZE3VIQTUQ, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.194795740 (769h5m12.065878785s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.194804257 (769h5m12.065887302s) [noise=true , address=145.79.227.29:4950 , piece_id=BMJ6KRI7QSHEIMHUATOXE2MBFIVMIWT4NJ2X6M2NHD52DQCQ3CRA, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195012674 (769h5m12.066095719s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.195021781 (769h5m12.066104846s) [noise=true , address=45.91.116.11:4149 , piece_id=MLPZD45I6N3V5PME2LNQTQPIGMDL6NVSJWMCT25BMST2MEX7GK5A, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195236690 (769h5m12.066319725s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:12.195244916 (769h5m12.066327951s) [noise=true , address=145.79.227.31:4801 , piece_id=6E2CFRSVOMAY566CZPYJUI7VZ7SRJHHZG4JFWSYKFIDRAWGJRSCA, seg_pos={0 0}, upload=672885]: Long tail cancellation due to optimal threshold
Jan 6 16:34:12.195274552 (769h5m12.066357597s) [seg_pos={0 0}, upload=672885]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 16:34:12.195381716 (769h5m12.066464761s) [upload=672885]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:12.195389681 (769h5m12.066472716s) [upload=672885]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:12.484471858 (769h5m12.355554903s) [upload=672885]: Done uploading segments...
Jan 6 16:34:12.484491015 (769h5m12.35557405s) [upload=672885]: Upload finished. err: <nil>
Duration: 1m0.002655359s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.X.......XX......................X.X....X.X.X....X.X...X....] duration:
23.780034401s trace:
https://jaeger.dev.storj.io/trace/2647644d1681d7b0
FAIL (2026-01-06 16:35:02 UTC)
orbiter warning: context deadline exceeded
Duration: 23.780061722s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.XX...X...X..X.XXXX..X.....XXX........X.......X..XX.....XX.X] duration:
30.002363755s trace:
https://jaeger.dev.storj.io/trace/36dd1cd45f4420cc
FAIL (2026-01-06 16:34:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ) context deadline exceeded; piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ) context deadline exceeded; piecestore: (Node ID: 1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, Piece ID: VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA) context deadline exceeded; piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ) context deadline exceeded
======LOGS======
Jan 6 16:34:23.725664194 (769h5m23.596747239s) [upload=672890]: Starting upload
Jan 6 16:34:23.725751641 (769h5m23.596834686s) [upload=672890]: Done starting upload
Jan 6 16:34:23.725958084 (769h5m23.59704112s) [upload=672890]: Uploading segments...
Jan 6 16:34:23.726102068 (769h5m23.597185104s) [upload=672890]: Got next segment. Inline: false
Jan 6 16:34:23.726139179 (769h5m23.597222214s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 16:34:23.726144118 (769h5m23.597227163s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 16:34:23.965881595 (769h5m23.83696463s) [seg_pos={0 0}, upload=672890]: Begin upload segment...
Jan 6 16:34:23.965936098 (769h5m23.837019143s) [seg_pos={0 0}, upload=672890]: Done begin upload segment.
Jan 6 16:34:23.965966356 (769h5m23.837049391s) [noise=true , address=145.79.227.6:4257 , piece_id=G3G5PW7TTBUUX43WRS27Q7IRERII2CSPOHSJF7CCW2JDS4SF76XA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.966002936 (769h5m23.837085971s) [noise=true , address=45.91.116.9:4262 , piece_id=2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.966726544 (769h5m23.837809579s) [noise=true , address=145.79.227.50:4375 , piece_id=UDJJ32G2LQWIENRTKANFOOXUGQHMGFFGNCVSIWHS74J72JVBL3EA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967009302 (769h5m23.838092347s) [noise=true , address=145.79.227.19:4079 , piece_id=RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967390548 (769h5m23.838473583s) [noise=true , address=145.79.227.28:4725 , piece_id=GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967734463 (769h5m23.838817498s) [noise=true , address=145.79.227.41:4086 , piece_id=VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.967991642 (769h5m23.839074677s) [noise=true , address=145.79.227.52:4447 , piece_id=FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968251918 (769h5m23.839334953s) [noise=true , address=145.79.227.13:4802 , piece_id=BIDGVMI2OHM6ATPAXETBJE4SBPWVXFA22YW3WEF7FI737JQEHOAA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968517413 (769h5m23.839600448s) [noise=true , address=145.79.227.14:4837 , piece_id=OA5CMTGROWVHFX4OK2IF5OIY2VQBA6WKUVEF523JDPMP3TKLRR7A, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.968781998 (769h5m23.839865033s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969032384 (769h5m23.840115419s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969281709 (769h5m23.840364754s) [noise=true , address=45.91.116.4:4939 , piece_id=WWM7Z4XQOOKV4AVX4WPGWDORYN6T4YQERLR2UHPHOR62JBJWP4NA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969552525 (769h5m23.84063556s) [noise=true , address=145.79.227.46:4601 , piece_id=VV7VWRLJ37DHIR7LAKUL3B5DMMYICZWYWLV6JZUDZUUXCTBROIUA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.969879137 (769h5m23.840962172s) [noise=true , address=145.79.227.25:4619 , piece_id=UHWNEWE6INQD7K4E75QFCWONUBHCVUH2HT2HOYROMBXY3YSXY5LA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.970130446 (769h5m23.841213481s) [noise=true , address=145.79.227.15:4876 , piece_id=2GEIIGCRC4KERAZOXNPCWUSYA2TZZQBU4SZ3SVK55OZJ6BIIUGWQ, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:23.970262667 (769h5m23.841345703s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Uploading piece...
Jan 6 16:34:25.774712127 (769h5m25.645795162s) [upload=672890]: Next returned nil segment
Jan 6 16:34:25.774729981 (769h5m25.645813046s) [upload=672890]: Waiting for error group managing segments...
Jan 6 16:34:26.137767310 (769h5m26.008850345s) [noise=true , address=145.79.227.52:4447 , piece_id=FHKYBLQBS2IU276LU5YR6QJF5MLYQUBJVJR4XD4BIG2AKNGM7L7A, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.162432296 (769h5m26.033515612s) [noise=true , address=145.79.227.28:4725 , piece_id=GGKGYNF4U27P3N733EFYP3L2W4ZY7U34U56SJAL3ILB6X6W2CPUQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.163113923 (769h5m26.034196958s) [noise=true , address=145.79.227.25:4619 , piece_id=UHWNEWE6INQD7K4E75QFCWONUBHCVUH2HT2HOYROMBXY3YSXY5LA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.193456581 (769h5m26.064539616s) [noise=true , address=145.79.227.19:4079 , piece_id=RELMPEAC2AP3YOSW7JPIRAPTMFRAURHZWNDJFIYDW5P52KZ2JOEQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.199003423 (769h5m26.070086468s) [noise=true , address=145.79.227.41:4086 , piece_id=VNND3QOTCXWRYJ4M5L4YHUMPF2FZML3URMDPDR7AEQFRO5EIJ4BA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.287764277 (769h5m26.158847343s) [noise=true , address=145.79.227.50:4375 , piece_id=UDJJ32G2LQWIENRTKANFOOXUGQHMGFFGNCVSIWHS74J72JVBL3EA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.327125958 (769h5m26.198208993s) [noise=true , address=145.79.227.14:4837 , piece_id=OA5CMTGROWVHFX4OK2IF5OIY2VQBA6WKUVEF523JDPMP3TKLRR7A, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.438912407 (769h5m26.309995442s) [noise=true , address=145.79.227.46:4601 , piece_id=VV7VWRLJ37DHIR7LAKUL3B5DMMYICZWYWLV6JZUDZUUXCTBROIUA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.496250988 (769h5m26.367334023s) [noise=true , address=45.91.116.4:4939 , piece_id=WWM7Z4XQOOKV4AVX4WPGWDORYN6T4YQERLR2UHPHOR62JBJWP4NA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.831068722 (769h5m26.702151768s) [noise=true , address=45.91.116.9:4262 , piece_id=2MLY4NHKF557AB3GFNBFWJO5XA2XIZ57N5VJKXKJCZDMKVIFOGLQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:26.996782847 (769h5m26.867866524s) [noise=true , address=145.79.227.13:4802 , piece_id=BIDGVMI2OHM6ATPAXETBJE4SBPWVXFA22YW3WEF7FI737JQEHOAA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.030623856 (769h5m26.901706911s) [noise=true , address=145.79.227.6:4257 , piece_id=G3G5PW7TTBUUX43WRS27Q7IRERII2CSPOHSJF7CCW2JDS4SF76XA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.219843789 (769h5m27.090926824s) [noise=true , address=145.79.227.15:4876 , piece_id=2GEIIGCRC4KERAZOXNPCWUSYA2TZZQBU4SZ3SVK55OZJ6BIIUGWQ, seg_pos={0 0}, upload=672890]: Done uploading piece. err: <nil>
Jan 6 16:34:27.219859249 (769h5m27.090942284s) [seg_pos={0 0}, upload=672890]: Segment reached optimal threshold of 13 pieces.
Jan 6 16:34:27.219990048 (769h5m27.091073093s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:27.219995408 (769h5m27.091078454s) [noise=true , address=145.79.227.48:4301 , piece_id=BUY5KNMOWZFJBZV4YTEPOD6XSEGCTQZNXFC5DHGRSODJ263AJBGA, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220223452 (769h5m27.091306487s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:27.220229183 (769h5m27.091312228s) [noise=true , address=45.91.116.11:4117 , piece_id=WMDLCERXWHN7AGIUFIWNSXQBJSWJYEKGLO64AZJEYIJVYGLHBEVA, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220717242 (769h5m27.091800277s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: write tcp 188.166.225.170:59640->145.79.227.47:4265: use of closed network connection; write tcp 188.166.225.170:59640->145.79.227.47:4265: use of closed network connection
Jan 6 16:34:27.220736199 (769h5m27.091819244s) [noise=true , address=145.79.227.47:4265 , piece_id=AK6Q7U4GXS5D5UT3QQQJTNVZRH7BEVHN6BWFKEVTTJLVGKCY5Z5Q, seg_pos={0 0}, upload=672890]: Long tail cancellation due to optimal threshold
Jan 6 16:34:27.220799980 (769h5m27.091883015s) [seg_pos={0 0}, upload=672890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 16:34:27.220910681 (769h5m27.091993716s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:27.220918155 (769h5m27.0920012s) [upload=672890]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:27.503741505 (769h5m27.37482454s) [upload=672890]: Done uploading segments...
Jan 6 16:34:27.503759960 (769h5m27.374842995s) [upload=672890]: Upload finished. err: <nil>
Duration: 30.002396398s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX.XX.XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration:
5.001246332s trace:
https://jaeger.dev.storj.io/trace/476d140c8d375e8f
FAIL (2026-01-06 16:34:46 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001268143s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
33.790099214s trace:
https://jaeger.dev.storj.io/trace/5986d31e6f10581d
FAIL (2026-01-06 16:34:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
53 , piece_id=FRLXXNNF2EC2UFPI24QSBJM4ITCBYEXQ3ESUZNZBASAVW6VLMDVQ, seg_pos={0 0}, upload=672884]: Uploading piece...
Jan 6 16:34:06.845131329 (769h5m6.716214364s) [noise=true , address=81.171.14.70:4041 , piece_id=JWNB5WWGJBP5QSZF7XTVT2HBPVHSRWLHOJYI2WKA74AH5PJB3KJQ, seg_pos={0 0}, upload=672884]: Uploading piece...
Jan 6 16:34:06.845138322 (769h5m6.716221367s) [noise=true , address=89.149.201.147:4016 , piece_id=OFQKR26R4CM6C7YXAQANWLWCKRYYSZ2GMS4C2ZEMYRNMD5GOMMPQ, seg_pos={0 0}, upload=672884]: Uploading piece...
Jan 6 16:34:06.845476576 (769h5m6.716559611s) [noise=true , address=185.167.204.206:4056 , piece_id=IYXFXKUAHP4DUNFDL3OZSSC5XNZP6FZNBQD2JACMNHQGEY2Y7EQA, seg_pos={0 0}, upload=672884]: Uploading piece...
Jan 6 16:34:06.845668391 (769h5m6.716751436s) [noise=true , address=185.167.204.205:4024 , piece_id=EPMS3XHLQWBZNDCGHZZKNVX4ZNXW3QYVEKRBENBX36VDNEXWLQ3A, seg_pos={0 0}, upload=672884]: Uploading piece...
Jan 6 16:34:06.845881247 (769h5m6.716964292s) [noise=true , address=81.171.14.70:4042 , piece_id=YNXHDZVPY2Y7K4MLYUYNFSTBPMJNY2SJOJBTGDSPCKDXULA3ILJQ, seg_pos={0 0}, upload=672884]: Uploading piece...
Jan 6 16:34:06.846153806 (769h5m6.717236841s) [noise=true , address=89.149.201.147:4014 , piece_id=2PDLVCW7PHPEU2Z3XOU7IIWNX6377UGMON2EX5WSVA6JAJAIDPRQ, seg_pos={0 0}, upload=672884]: Uploading piece...
Jan 6 16:34:06.846325272 (769h5m6.717408307s) [noise=true , address=81.171.14.70:4040 , piece_id=OZ4PTH4ZUF23CN3DDCI5NDI4ICJKAH7K2I2URXABW6HVOJQCBLBA, seg_pos={0 0}, upload=672884]: Uploading piece...
Jan 6 16:34:06.846832628 (769h5m6.717915663s) [noise=true , address=81.171.14.70:4032 , piece_id=FBG6SQNSW7WMQFZQDFTXB2IX2N2Q3ETIM3JTMRDVCG3ABV7RMGTA, seg_pos={0 0}, upload=672884]: Uploading piece...
Jan 6 16:34:06.847350634 (769h5m6.718433669s) [noise=true , address=89.149.201.147:4022 , piece_id=DRVZGIWRF5PNXWWGY5UDN7PONJC2OPS4KACP2WULSR4BY7HYWQNA, seg_pos={0 0}, upload=672884]: Uploading piece...
Jan 6 16:34:06.847748132 (769h5m6.718831177s) [noise=true , address=185.167.204.205:4022 , piece_id=T23Q5N6UWWTIQKFDJJDKYZHO3RTTUSM2I7YQYNP4XPEPX2ZTY2MQ, seg_pos={0 0}, upload=672884]: Uploading piece...
Jan 6 16:34:07.672772494 (769h5m7.543855529s) [upload=672884]: Next returned nil segment
Jan 6 16:34:07.672787222 (769h5m7.543870267s) [upload=672884]: Waiting for error group managing segments...
Jan 6 16:34:08.065167634 (769h5m7.936250669s) [noise=true , address=185.167.204.205:4004 , piece_id=YHSBKVTFNXMFJRDVA7BES5KJB7O44UUTD47PTUPR25OEAWOBHWWA, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.068462809 (769h5m7.939545854s) [noise=true , address=185.167.204.205:4024 , piece_id=EPMS3XHLQWBZNDCGHZZKNVX4ZNXW3QYVEKRBENBX36VDNEXWLQ3A, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.069627877 (769h5m7.940710922s) [noise=true , address=185.167.204.206:4057 , piece_id=PT6F7K2OTD6T3DJ33YYZEHOITDZCQWFGVU667LTMTJFVPS7AAWGQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.070241265 (769h5m7.941324301s) [noise=true , address=185.167.204.206:4056 , piece_id=IYXFXKUAHP4DUNFDL3OZSSC5XNZP6FZNBQD2JACMNHQGEY2Y7EQA, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.070408644 (769h5m7.941491689s) [noise=true , address=185.167.204.205:4013 , piece_id=5HWT4EHPKAH2XL7QXSAT23Z7EQY5VWHABSXMO24N5QZOZQ6TAIIQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.071976901 (769h5m7.943059936s) [noise=true , address=185.167.204.206:4054 , piece_id=WIW3IHXHEJ4DD5D5PXDZXVIJOOUBZTB677EMM4O25FIUAVAWLNVA, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.074259466 (769h5m7.945342511s) [noise=true , address=185.167.204.206:4053 , piece_id=FRLXXNNF2EC2UFPI24QSBJM4ITCBYEXQ3ESUZNZBASAVW6VLMDVQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.098452775 (769h5m7.96953581s) [noise=true , address=185.167.204.206:4037 , piece_id=ZAJ7D2I273A4HT2S5RMUJE3LVZU3ESBA7PR73UQVC2FDDPMKPFCA, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.756842320 (769h5m8.627925355s) [noise=true , address=81.171.14.70:4042 , piece_id=YNXHDZVPY2Y7K4MLYUYNFSTBPMJNY2SJOJBTGDSPCKDXULA3ILJQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.757639067 (769h5m8.628722102s) [noise=true , address=81.171.14.70:4032 , piece_id=FBG6SQNSW7WMQFZQDFTXB2IX2N2Q3ETIM3JTMRDVCG3ABV7RMGTA, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.781245487 (769h5m8.652328533s) [noise=true , address=81.171.14.70:4030 , piece_id=CAOSJPPLG52EHN7D46HI6A2PSBBD4UAJUBWM32WNMNUW6GL5R3VQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.783479261 (769h5m8.654562296s) [noise=true , address=81.171.14.70:4036 , piece_id=WYNAAEK73IQFGLS43SPIMB335STYZNOL2YEEFPIFP2R3XE4KZOGA, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.825133627 (769h5m8.696216662s) [noise=true , address=81.171.14.70:4041 , piece_id=JWNB5WWGJBP5QSZF7XTVT2HBPVHSRWLHOJYI2WKA74AH5PJB3KJQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:08.843774993 (769h5m8.714858028s) [noise=true , address=81.171.14.70:4040 , piece_id=OZ4PTH4ZUF23CN3DDCI5NDI4ICJKAH7K2I2URXABW6HVOJQCBLBA, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:09.279642953 (769h5m9.150725998s) [noise=true , address=81.171.14.70:4046 , piece_id=FMRNEA737K73BJN56ZLIV2S4VR4I5UG6NU6S2II2ATQ6VUVRG5TQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:09.382503756 (769h5m9.253586791s) [noise=true , address=81.171.14.70:4029 , piece_id=FSLEVYRJQQILDQA6UUUQG42RHTF7BTKMZK7FC7ROEE2D45JLBV2A, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:09.427374096 (769h5m9.298457141s) [noise=true , address=89.149.201.147:4016 , piece_id=OFQKR26R4CM6C7YXAQANWLWCKRYYSZ2GMS4C2ZEMYRNMD5GOMMPQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:09.441948036 (769h5m9.313031071s) [noise=true , address=89.149.201.147:4007 , piece_id=J2H3L5Z5IZNCG7RUDRTXELT4NSTY6EHYD4L763WI5HY2FZAJXJZA, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:09.519377915 (769h5m9.39046095s) [noise=true , address=89.149.201.147:4003 , piece_id=363CEXOU42IP56ULIGXILXGJYMVHE5XK4Y5E7YKVNGTJ7K3ZLHXQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:09.550575661 (769h5m9.421658706s) [noise=true , address=89.149.201.147:4011 , piece_id=CZJ462FAJZSQAFGXX3USP3JLP63ZGGZYELOWPB2ZPNGNUTXDHIYQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:09.654787014 (769h5m9.525870059s) [noise=true , address=89.149.201.147:4022 , piece_id=DRVZGIWRF5PNXWWGY5UDN7PONJC2OPS4KACP2WULSR4BY7HYWQNA, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:09.968920414 (769h5m9.840003449s) [noise=true , address=89.149.201.147:4001 , piece_id=RA4E6JOJWIKTFNPG2RDLZNYVCRHFQFRAG4W77SGHJJPRSLDVDYNQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:10.526394569 (769h5m10.397477604s) [noise=true , address=185.167.204.206:4072 , piece_id=4VPDHKOA6GKBGTL35B2J7JH7IWUDUU774QGZTMBX2MGC7ID3OG2Q, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:11.161994769 (769h5m11.033077804s) [noise=true , address=89.149.201.147:4014 , piece_id=2PDLVCW7PHPEU2Z3XOU7IIWNX6377UGMON2EX5WSVA6JAJAIDPRQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:12.830614086 (769h5m12.701697121s) [noise=true , address=185.167.204.205:4014 , piece_id=5NQ2DS3XNDLXUWSXXRN45JFMKF7EA6VEVIZPCVA7J4YDU6P3HA7A, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:13.054408365 (769h5m12.9254914s) [noise=true , address=185.167.204.206:4046 , piece_id=ZQQQM42FCOB7ROZKAJV7SI76VR35LSHBUUL2BICQD2G65OMG6JYQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: <nil>
Jan 6 16:34:13.054442550 (769h5m12.925525585s) [seg_pos={0 0}, upload=672884]: Segment reached optimal threshold of 26 pieces.
Jan 6 16:34:13.054708998 (769h5m12.925792033s) [noise=true , address=185.167.204.205:4022 , piece_id=T23Q5N6UWWTIQKFDJJDKYZHO3RTTUSM2I7YQYNP4XPEPX2ZTY2MQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:13.054717694 (769h5m12.925800739s) [noise=true , address=185.167.204.205:4022 , piece_id=T23Q5N6UWWTIQKFDJJDKYZHO3RTTUSM2I7YQYNP4XPEPX2ZTY2MQ, seg_pos={0 0}, upload=672884]: Long tail cancellation due to optimal threshold
Jan 6 16:34:13.055049826 (769h5m12.926132871s) [noise=true , address=185.167.204.206:4067 , piece_id=7FEFQN727E6P6VDEQXK25WNDZZ47LAK52GD25B2O5XIVYEPIB2HQ, seg_pos={0 0}, upload=672884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS): protocol: expected piece hash; context canceled; EOF
Jan 6 16:34:13.055081166 (769h5m12.926164201s) [noise=true , address=185.167.204.206:4067 , piece_id=7FEFQN727E6P6VDEQXK25WNDZZ47LAK52GD25B2O5XIVYEPIB2HQ, seg_pos={0 0}, upload=672884]: Long tail cancellation due to optimal threshold
Jan 6 16:34:13.055111614 (769h5m12.926194659s) [seg_pos={0 0}, upload=672884]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Jan 6 16:34:13.055209641 (769h5m12.926292676s) [upload=672884]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:34:13.055214991 (769h5m12.926298026s) [upload=672884]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:34:13.292364800 (769h5m13.163447865s) [upload=672884]: Done uploading segments...
Jan 6 16:34:13.292383075 (769h5m13.16346612s) [upload=672884]: Upload finished. err: <nil>
Duration: 33.790147105s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.....X.....X......................X......X..................] duration:
22.380921782s trace:
https://jaeger.dev.storj.io/trace/d5e2d8b8127d900
FAIL (2026-01-06 16:34:28 UTC)
orbiter warning: context deadline exceeded
Duration: 22.380977859s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..X.......XX......................X.X....X.X.X....X.X...X...] duration:
28.962473323s trace:
https://jaeger.dev.storj.io/trace/2647644d1681d7b0
FAIL (2026-01-06 16:34:10 UTC)
orbiter warning: context deadline exceeded
Duration: 28.962496718s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
10 out of 60 is failed (threshold=3) [...........X......X..X.X.X........X.....XX......X..........X] duration:
1m0.002868304s trace:
https://jaeger.dev.storj.io/trace/6d20c35f6e8b377e
FAIL (2026-01-06 16:34:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
.12:4797 , piece_id=VRNPAO75MP25VZGXFRK47J4SDEOJYJKEVH34QTH5QLRVPOLY7HAQ, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.568735340 (769h4m7.439818395s) [noise=true , address=45.91.116.11:4137 , piece_id=KRFAKXM4LJKEOG44XBV25TWQBQR66H6DZNTDIZTGTOMQQKJKRUAA, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.569169016 (769h4m7.440252071s) [noise=true , address=145.79.227.36:4524 , piece_id=5CEVJAMTVQ3SKWOQVGDQY3XBK4I3W5BZHEXJQMGV3YFC4T3SZ5UQ, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.569569368 (769h4m7.440652413s) [noise=true , address=145.79.227.18:4008 , piece_id=LN6UMXT6WE4ZIHRHYLQ72J3GDD7SDPLK7QFV7YNDXIPE2LJCXGPA, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.569976213 (769h4m7.441059258s) [noise=true , address=145.79.227.52:4475 , piece_id=TB3DGE3DMTBRDBMWXW3TGHOIXDSR2OAJFXLKRXTZDC7LF5YZ5SZQ, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.570383329 (769h4m7.441466374s) [noise=true , address=145.79.227.30:4796 , piece_id=K2KI4YJ4W7DPS6PLW3BNPXF46THJTYF2CCVY4UUAYR7QRSQLQCBA, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.570590243 (769h4m7.441673288s) [noise=true , address=145.79.227.43:4182 , piece_id=QOOPFO5K5USAIYQ6Q2AHOJRIMUFWI46TYVQOXRW6DLCTTZ52QJTA, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.571080737 (769h4m7.442163792s) [noise=true , address=145.79.227.44:4227 , piece_id=WRDXIG54YSEJRYNHTFVNORJDK72PTMCOVW5LWQNL53YCAAN6B7TA, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.571511987 (769h4m7.442595033s) [noise=true , address=145.79.227.5:4225 , piece_id=6PPGWFOEODK4GDLQG42ROQHFS4PTG3SOMJYUDNHKKEQ5KNZ2T5EA, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.571883345 (769h4m7.44296639s) [noise=true , address=145.79.227.16:4924 , piece_id=HLLLQG7OJ7S726BJSCHHB4V3PPHJNA2RDA77ZSAAYSBICFJB65PQ, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.572323523 (769h4m7.443406569s) [noise=true , address=145.79.227.29:4965 , piece_id=YILLHYUTF6DWGJN237AUQHZ2GXCZVGJRW6JRFI3TIXLNA75EB6LA, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.572714538 (769h4m7.443797593s) [noise=true , address=45.91.116.6:4001 , piece_id=XB4L7UIAZZGYQJTYL574RGL6NFJTYWGNQYIUUKO7ABPXDOV6PXUQ, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.573184111 (769h4m7.444267157s) [noise=true , address=77.90.31.17:4802 , piece_id=R7YAXRAFZWF6ICBFPR3KRLWNUU73K4KVIGHM3Z7RQR5KYV3A7QTA, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.573277470 (769h4m7.444360515s) [noise=true , address=64.49.13.137:4501 , piece_id=4DN4XMUXTP4A5QGVUCHUIXI27SNA5MQOJVQSTL36HEOYXFFICAZA, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.573774335 (769h4m7.44485738s) [noise=true , address=45.91.116.9:4292 , piece_id=JIG3SN2XUDVELNTT2BZPBEKTIKSRJEC6VKW3QJDOLEHFZNEHLEEA, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:07.574288074 (769h4m7.445371119s) [noise=true , address=64.49.13.154:4933 , piece_id=77CNRB62UAZBAJCJABDHSVGBCIP5LBWQYMXONVNJEZNKPHKZSYFQ, seg_pos={0 0}, upload=672871]: Uploading piece...
Jan 6 16:33:08.990693848 (769h4m8.861776883s) [upload=672871]: Next returned nil segment
Jan 6 16:33:08.990707063 (769h4m8.861790108s) [upload=672871]: Waiting for error group managing segments...
Jan 6 16:33:09.368219595 (769h4m9.23930263s) [noise=true , address=64.49.13.154:4933 , piece_id=77CNRB62UAZBAJCJABDHSVGBCIP5LBWQYMXONVNJEZNKPHKZSYFQ, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:09.394068506 (769h4m9.265151551s) [noise=true , address=64.49.13.137:4501 , piece_id=4DN4XMUXTP4A5QGVUCHUIXI27SNA5MQOJVQSTL36HEOYXFFICAZA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:09.417418006 (769h4m9.288501051s) [noise=true , address=145.79.227.52:4475 , piece_id=TB3DGE3DMTBRDBMWXW3TGHOIXDSR2OAJFXLKRXTZDC7LF5YZ5SZQ, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:09.431962562 (769h4m9.303045607s) [noise=true , address=145.79.227.16:4924 , piece_id=HLLLQG7OJ7S726BJSCHHB4V3PPHJNA2RDA77ZSAAYSBICFJB65PQ, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:09.444410404 (769h4m9.315493439s) [noise=true , address=145.79.227.36:4524 , piece_id=5CEVJAMTVQ3SKWOQVGDQY3XBK4I3W5BZHEXJQMGV3YFC4T3SZ5UQ, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:09.461892613 (769h4m9.332975658s) [noise=true , address=145.79.227.18:4008 , piece_id=LN6UMXT6WE4ZIHRHYLQ72J3GDD7SDPLK7QFV7YNDXIPE2LJCXGPA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:09.467475885 (769h4m9.33855893s) [noise=true , address=145.79.227.5:4225 , piece_id=6PPGWFOEODK4GDLQG42ROQHFS4PTG3SOMJYUDNHKKEQ5KNZ2T5EA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:09.551266705 (769h4m9.42234974s) [noise=true , address=145.79.227.12:4797 , piece_id=VRNPAO75MP25VZGXFRK47J4SDEOJYJKEVH34QTH5QLRVPOLY7HAQ, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:10.226743336 (769h4m10.097826371s) [noise=true , address=45.91.116.11:4137 , piece_id=KRFAKXM4LJKEOG44XBV25TWQBQR66H6DZNTDIZTGTOMQQKJKRUAA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:10.268206499 (769h4m10.139289534s) [noise=true , address=45.91.116.6:4001 , piece_id=XB4L7UIAZZGYQJTYL574RGL6NFJTYWGNQYIUUKO7ABPXDOV6PXUQ, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:10.312361967 (769h4m10.183445002s) [noise=true , address=145.79.227.30:4796 , piece_id=K2KI4YJ4W7DPS6PLW3BNPXF46THJTYF2CCVY4UUAYR7QRSQLQCBA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:10.323176990 (769h4m10.194260035s) [noise=true , address=145.79.227.43:4182 , piece_id=QOOPFO5K5USAIYQ6Q2AHOJRIMUFWI46TYVQOXRW6DLCTTZ52QJTA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:10.355932101 (769h4m10.227015136s) [noise=true , address=77.90.31.17:4802 , piece_id=R7YAXRAFZWF6ICBFPR3KRLWNUU73K4KVIGHM3Z7RQR5KYV3A7QTA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:10.381590659 (769h4m10.252673704s) [noise=true , address=145.79.227.44:4227 , piece_id=WRDXIG54YSEJRYNHTFVNORJDK72PTMCOVW5LWQNL53YCAAN6B7TA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: <nil>
Jan 6 16:33:10.381608664 (769h4m10.252691699s) [seg_pos={0 0}, upload=672871]: Segment reached optimal threshold of 14 pieces.
Jan 6 16:33:10.381821619 (769h4m10.252904654s) [noise=true , address=145.79.227.29:4965 , piece_id=YILLHYUTF6DWGJN237AUQHZ2GXCZVGJRW6JRFI3TIXLNA75EB6LA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ToFTqjRAWvpRQzf6vyHKDpforPyb4LbDGcCVhRhkwiBGKihnD): protocol: expected piece hash; context canceled; EOF
Jan 6 16:33:10.381828322 (769h4m10.252911367s) [noise=true , address=145.79.227.29:4965 , piece_id=YILLHYUTF6DWGJN237AUQHZ2GXCZVGJRW6JRFI3TIXLNA75EB6LA, seg_pos={0 0}, upload=672871]: Long tail cancellation due to optimal threshold
Jan 6 16:33:10.382008645 (769h4m10.25309168s) [noise=true , address=145.79.227.33:4892 , piece_id=3CIFBIKTS2V5ADIQETRMQKVUEHWLQAPIRCV3NCP6I57NBFFW5ECA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5): protocol: expected piece hash; context canceled; EOF
Jan 6 16:33:10.382016530 (769h4m10.253099585s) [noise=true , address=145.79.227.33:4892 , piece_id=3CIFBIKTS2V5ADIQETRMQKVUEHWLQAPIRCV3NCP6I57NBFFW5ECA, seg_pos={0 0}, upload=672871]: Long tail cancellation due to optimal threshold
Jan 6 16:33:10.382062979 (769h4m10.253146024s) [noise=true , address=145.79.227.4:4205 , piece_id=DPTNPVDQ6WWLXSGWGOPUH5L5TIWF3DKI6LPFXZSM62ULCN3KTIYA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: write tcp 188.166.225.170:54386->145.79.227.4:4205: use of closed network connection; write tcp 188.166.225.170:54386->145.79.227.4:4205: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54386->145.79.227.4:4205: use of closed network connection
Jan 6 16:33:10.382097875 (769h4m10.25318092s) [noise=true , address=145.79.227.4:4205 , piece_id=DPTNPVDQ6WWLXSGWGOPUH5L5TIWF3DKI6LPFXZSM62ULCN3KTIYA, seg_pos={0 0}, upload=672871]: Long tail cancellation due to optimal threshold
Jan 6 16:33:10.382270795 (769h4m10.25335383s) [noise=true , address=45.91.116.9:4292 , piece_id=JIG3SN2XUDVELNTT2BZPBEKTIKSRJEC6VKW3QJDOLEHFZNEHLEEA, seg_pos={0 0}, upload=672871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tnWSKAebAwLeGWGRsNnYpcWgbX5UHSVRUnnu9qesUZCK4vY81): protocol: read tcp 188.166.225.170:45108->45.91.116.9:4292: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 16:33:10.382277177 (769h4m10.253360212s) [noise=true , address=45.91.116.9:4292 , piece_id=JIG3SN2XUDVELNTT2BZPBEKTIKSRJEC6VKW3QJDOLEHFZNEHLEEA, seg_pos={0 0}, upload=672871]: Long tail cancellation due to optimal threshold
Jan 6 16:33:10.382294198 (769h4m10.253377233s) [seg_pos={0 0}, upload=672871]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 16:33:10.382364482 (769h4m10.253447517s) [upload=672871]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 16:33:10.382369171 (769h4m10.253452216s) [upload=672871]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 16:33:10.676756752 (769h4m10.547839787s) [upload=672871]: Done uploading segments...
Jan 6 16:33:10.677393374 (769h4m10.548476429s) [upload=672871]: Upload finished. err: <nil>
Duration: 1m0.00289238s
core
uplink
us-select-1
us1-placement-20
|