Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 14.154054563s trace: https://jaeger.dev.storj.io/trace/2c300281cb8f70aa FAIL (2025-12-27 00:32:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== {0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments... Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces. Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments... Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil> Duration: 14.154069721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.005390007s trace: https://jaeger.dev.storj.io/trace/2c300281cb8f70aa FAIL (2025-12-27 00:32:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== {0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece... Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments... Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil> Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces. Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments... Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil> Duration: 1m0.005402432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.XX...X.........X.........X.......X.XX.XX.XXXX..X.XXX.X.X..] duration: 22.897194425s trace: https://jaeger.dev.storj.io/trace/4eb1a943ce14edee FAIL (2025-12-27 00:31:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: LZXI3WT4WFQYWG244GO7N7X6BAUU7QKKMIRHEDMJUCLGVDPIAGBQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: GIXFWJNOMXONQCQIONLZNHZUKB65JL6VVPJ5CNWKUONQW25VRYMQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: M7FXCTF3JOEBEMMAMH5GP3HLW3OLXRD26HQLJOC2PVQT7VZUECPA) context deadline exceeded ======LOGS====== 29 (513h0m33.172672684s) [upload=450154]: Done starting upload Dec 27 00:29:33.301683027 (513h0m33.172766082s) [upload=450154]: Uploading segments... Dec 27 00:29:33.301943613 (513h0m33.173026668s) [upload=450154]: Got next segment. Inline: false Dec 27 00:29:33.302010490 (513h0m33.173093555s) [upload=450154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:29:33.302016772 (513h0m33.173099807s) [upload=450154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:29:33.550475352 (513h0m33.421558387s) [seg_pos={0 0}, upload=450154]: Begin upload segment... Dec 27 00:29:33.550519195 (513h0m33.42160223s) [seg_pos={0 0}, upload=450154]: Done begin upload segment. Dec 27 00:29:33.550544744 (513h0m33.421627779s) [noise=true , address=145.79.227.6:4258 , piece_id=2Y3M2TY37O72IJM7GRAUPACI5FG2OIIYPFBPIHSB3KGIOSFZQXXQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.550611070 (513h0m33.421694105s) [noise=true , address=145.79.227.2:4113 , piece_id=KAKUAAKICZDRBTJRWYMZ2WFF623OBFYT65J6ESSSHSCKTYIK32MQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.551163252 (513h0m33.422246287s) [noise=true , address=145.79.227.47:4266 , piece_id=FAAG3Z5EBOMWKISWHUHFSZX2KTLVTIGEZRBEXWQ4FVQDKTWURDAQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.551526693 (513h0m33.422609728s) [noise=true , address=77.90.31.12:4589 , piece_id=HBJAIDCRKDGXVGBXIOWLWLFQ5UHOJKBYOBKWXNCZTCPFSJ6SXSNA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.551970970 (513h0m33.423054005s) [noise=true , address=145.79.227.51:4411 , piece_id=XPETBF2G6MVQ3VMGTX6ZG2QFS5555XKPSGCUXFQ57RG65UWH76KA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.552216928 (513h0m33.423299963s) [noise=true , address=45.91.116.9:4264 , piece_id=IVBP4NTK4VY4EG7R3AKGFU662OO54I46XOFYUKCB6EN2SXSJDRSQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.552513452 (513h0m33.423596487s) [noise=true , address=77.90.31.18:4808 , piece_id=C7NBOPC5GZ53KSG5U3MJW5ARCQPVK7XHYDBQNA642KNP4HGQF2HA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.552769530 (513h0m33.423852565s) [noise=true , address=45.91.116.14:4192 , piece_id=HBJ4PCTJABCHSHL53PUQWFEINL4BRTYBO3AGC544ZCMSLW6325FQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553043312 (513h0m33.424126357s) [noise=true , address=45.91.116.6:4974 , piece_id=2P4WX7V5SKN42QYPGLPFLN26IC5REVXSQ7RGZPGDPUH5VWV2Y5CA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553320239 (513h0m33.424403284s) [noise=true , address=145.79.227.37:4567 , piece_id=DPCYHB5ONQ7DGYFDQENSRILA7AAFDVO2OVKRT6X32VSOJAKQ5CJQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553591014 (513h0m33.424674059s) [noise=true , address=45.91.116.13:4155 , piece_id=LKJZYTNJZPUUOG3OODPM6TGMYY4WTLFAOGGC2SBCCIHG35WBTPBQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553891426 (513h0m33.424974461s) [noise=true , address=45.91.116.10:4081 , piece_id=LZXI3WT4WFQYWG244GO7N7X6BAUU7QKKMIRHEDMJUCLGVDPIAGBQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.554070657 (513h0m33.425153692s) [noise=true , address=77.90.31.13:4629 , piece_id=GIXFWJNOMXONQCQIONLZNHZUKB65JL6VVPJ5CNWKUONQW25VRYMQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.554466571 (513h0m33.425549606s) [noise=true , address=145.79.227.15:4875 , piece_id=RSURGRNZPY3KNRXPYBRRF2AKOW4CW4WVBDEJGH5ZFAC5KJRUCB4A, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.554784676 (513h0m33.425867712s) [noise=true , address=77.90.31.10:4519 , piece_id=7ZQTTHLJCKSN55EJNCX3TD4YUSIE5EI2DOUKT2RFWOT4UMEMOTKA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.555093044 (513h0m33.426176089s) [noise=true , address=45.91.116.5:4299 , piece_id=M7FXCTF3JOEBEMMAMH5GP3HLW3OLXRD26HQLJOC2PVQT7VZUECPA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:35.261344284 (513h0m35.132427329s) [upload=450154]: Next returned nil segment Dec 27 00:29:35.261359944 (513h0m35.132442979s) [upload=450154]: Waiting for error group managing segments... Dec 27 00:29:35.553544942 (513h0m35.424627977s) [noise=true , address=145.79.227.6:4258 , piece_id=2Y3M2TY37O72IJM7GRAUPACI5FG2OIIYPFBPIHSB3KGIOSFZQXXQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.558212230 (513h0m35.429295275s) [noise=true , address=145.79.227.2:4113 , piece_id=KAKUAAKICZDRBTJRWYMZ2WFF623OBFYT65J6ESSSHSCKTYIK32MQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.560005734 (513h0m35.431088779s) [noise=true , address=145.79.227.37:4567 , piece_id=DPCYHB5ONQ7DGYFDQENSRILA7AAFDVO2OVKRT6X32VSOJAKQ5CJQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.560447175 (513h0m35.43153021s) [noise=true , address=145.79.227.15:4875 , piece_id=RSURGRNZPY3KNRXPYBRRF2AKOW4CW4WVBDEJGH5ZFAC5KJRUCB4A, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.560800788 (513h0m35.431883823s) [noise=true , address=145.79.227.51:4411 , piece_id=XPETBF2G6MVQ3VMGTX6ZG2QFS5555XKPSGCUXFQ57RG65UWH76KA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.583437762 (513h0m35.454520797s) [noise=true , address=145.79.227.47:4266 , piece_id=FAAG3Z5EBOMWKISWHUHFSZX2KTLVTIGEZRBEXWQ4FVQDKTWURDAQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.743623151 (513h0m35.614706196s) [noise=true , address=77.90.31.18:4808 , piece_id=C7NBOPC5GZ53KSG5U3MJW5ARCQPVK7XHYDBQNA642KNP4HGQF2HA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.754063620 (513h0m35.625146665s) [noise=true , address=77.90.31.13:4629 , piece_id=GIXFWJNOMXONQCQIONLZNHZUKB65JL6VVPJ5CNWKUONQW25VRYMQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.902109343 (513h0m35.773192388s) [noise=true , address=45.91.116.6:4974 , piece_id=2P4WX7V5SKN42QYPGLPFLN26IC5REVXSQ7RGZPGDPUH5VWV2Y5CA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.903974245 (513h0m35.77505728s) [noise=true , address=45.91.116.13:4155 , piece_id=LKJZYTNJZPUUOG3OODPM6TGMYY4WTLFAOGGC2SBCCIHG35WBTPBQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.904311547 (513h0m35.775394582s) [noise=true , address=45.91.116.10:4081 , piece_id=LZXI3WT4WFQYWG244GO7N7X6BAUU7QKKMIRHEDMJUCLGVDPIAGBQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.905333052 (513h0m35.776416087s) [noise=true , address=45.91.116.9:4264 , piece_id=IVBP4NTK4VY4EG7R3AKGFU662OO54I46XOFYUKCB6EN2SXSJDRSQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:36.168150111 (513h0m36.039233156s) [noise=true , address=45.91.116.5:4299 , piece_id=M7FXCTF3JOEBEMMAMH5GP3HLW3OLXRD26HQLJOC2PVQT7VZUECPA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:36.168171072 (513h0m36.039254107s) [seg_pos={0 0}, upload=450154]: Segment reached optimal threshold of 13 pieces. Dec 27 00:29:36.168343159 (513h0m36.039426204s) [noise=true , address=77.90.31.12:4589 , piece_id=HBJAIDCRKDGXVGBXIOWLWLFQ5UHOJKBYOBKWXNCZTCPFSJ6SXSNA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:49956->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:49956->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49956->77.90.31.12:4589: use of closed network connection Dec 27 00:29:36.168350814 (513h0m36.039433849s) [noise=true , address=77.90.31.12:4589 , piece_id=HBJAIDCRKDGXVGBXIOWLWLFQ5UHOJKBYOBKWXNCZTCPFSJ6SXSNA, seg_pos={0 0}, upload=450154]: Long tail cancellation due to optimal threshold Dec 27 00:29:36.168508604 (513h0m36.039591639s) [noise=true , address=45.91.116.14:4192 , piece_id=HBJ4PCTJABCHSHL53PUQWFEINL4BRTYBO3AGC544ZCMSLW6325FQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; write tcp 188.166.225.170:53916->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:53916->45.91.116.14:4192: use of closed network connection; EOF Dec 27 00:29:36.168524434 (513h0m36.039607469s) [noise=true , address=45.91.116.14:4192 , piece_id=HBJ4PCTJABCHSHL53PUQWFEINL4BRTYBO3AGC544ZCMSLW6325FQ, seg_pos={0 0}, upload=450154]: Long tail cancellation due to optimal threshold Dec 27 00:29:36.168617320 (513h0m36.039700356s) [noise=true , address=77.90.31.10:4519 , piece_id=7ZQTTHLJCKSN55EJNCX3TD4YUSIE5EI2DOUKT2RFWOT4UMEMOTKA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 00:29:36.168624264 (513h0m36.0397073s) [noise=true , address=77.90.31.10:4519 , piece_id=7ZQTTHLJCKSN55EJNCX3TD4YUSIE5EI2DOUKT2RFWOT4UMEMOTKA, seg_pos={0 0}, upload=450154]: Long tail cancellation due to optimal threshold Dec 27 00:29:36.168640325 (513h0m36.03972336s) [seg_pos={0 0}, upload=450154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:29:36.168731549 (513h0m36.039814584s) [upload=450154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:29:36.168737630 (513h0m36.039820665s) [upload=450154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:29:36.458582313 (513h0m36.329665348s) [upload=450154]: Done uploading segments... Dec 27 00:29:36.458601128 (513h0m36.329684164s) [upload=450154]: Upload finished. err: <nil> Duration: 22.897227998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.XX...X.........X.........X.......X.XX.XX.XXXX..X.XXX.X.X.] duration: 6.000799478s trace: https://jaeger.dev.storj.io/trace/4eb1a943ce14edee FAIL (2025-12-27 00:30:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: LZXI3WT4WFQYWG244GO7N7X6BAUU7QKKMIRHEDMJUCLGVDPIAGBQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: GIXFWJNOMXONQCQIONLZNHZUKB65JL6VVPJ5CNWKUONQW25VRYMQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: M7FXCTF3JOEBEMMAMH5GP3HLW3OLXRD26HQLJOC2PVQT7VZUECPA) context deadline exceeded ======LOGS====== 29 (513h0m33.172672684s) [upload=450154]: Done starting upload Dec 27 00:29:33.301683027 (513h0m33.172766082s) [upload=450154]: Uploading segments... Dec 27 00:29:33.301943613 (513h0m33.173026668s) [upload=450154]: Got next segment. Inline: false Dec 27 00:29:33.302010490 (513h0m33.173093555s) [upload=450154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:29:33.302016772 (513h0m33.173099807s) [upload=450154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:29:33.550475352 (513h0m33.421558387s) [seg_pos={0 0}, upload=450154]: Begin upload segment... Dec 27 00:29:33.550519195 (513h0m33.42160223s) [seg_pos={0 0}, upload=450154]: Done begin upload segment. Dec 27 00:29:33.550544744 (513h0m33.421627779s) [noise=true , address=145.79.227.6:4258 , piece_id=2Y3M2TY37O72IJM7GRAUPACI5FG2OIIYPFBPIHSB3KGIOSFZQXXQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.550611070 (513h0m33.421694105s) [noise=true , address=145.79.227.2:4113 , piece_id=KAKUAAKICZDRBTJRWYMZ2WFF623OBFYT65J6ESSSHSCKTYIK32MQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.551163252 (513h0m33.422246287s) [noise=true , address=145.79.227.47:4266 , piece_id=FAAG3Z5EBOMWKISWHUHFSZX2KTLVTIGEZRBEXWQ4FVQDKTWURDAQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.551526693 (513h0m33.422609728s) [noise=true , address=77.90.31.12:4589 , piece_id=HBJAIDCRKDGXVGBXIOWLWLFQ5UHOJKBYOBKWXNCZTCPFSJ6SXSNA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.551970970 (513h0m33.423054005s) [noise=true , address=145.79.227.51:4411 , piece_id=XPETBF2G6MVQ3VMGTX6ZG2QFS5555XKPSGCUXFQ57RG65UWH76KA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.552216928 (513h0m33.423299963s) [noise=true , address=45.91.116.9:4264 , piece_id=IVBP4NTK4VY4EG7R3AKGFU662OO54I46XOFYUKCB6EN2SXSJDRSQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.552513452 (513h0m33.423596487s) [noise=true , address=77.90.31.18:4808 , piece_id=C7NBOPC5GZ53KSG5U3MJW5ARCQPVK7XHYDBQNA642KNP4HGQF2HA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.552769530 (513h0m33.423852565s) [noise=true , address=45.91.116.14:4192 , piece_id=HBJ4PCTJABCHSHL53PUQWFEINL4BRTYBO3AGC544ZCMSLW6325FQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553043312 (513h0m33.424126357s) [noise=true , address=45.91.116.6:4974 , piece_id=2P4WX7V5SKN42QYPGLPFLN26IC5REVXSQ7RGZPGDPUH5VWV2Y5CA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553320239 (513h0m33.424403284s) [noise=true , address=145.79.227.37:4567 , piece_id=DPCYHB5ONQ7DGYFDQENSRILA7AAFDVO2OVKRT6X32VSOJAKQ5CJQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553591014 (513h0m33.424674059s) [noise=true , address=45.91.116.13:4155 , piece_id=LKJZYTNJZPUUOG3OODPM6TGMYY4WTLFAOGGC2SBCCIHG35WBTPBQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553891426 (513h0m33.424974461s) [noise=true , address=45.91.116.10:4081 , piece_id=LZXI3WT4WFQYWG244GO7N7X6BAUU7QKKMIRHEDMJUCLGVDPIAGBQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.554070657 (513h0m33.425153692s) [noise=true , address=77.90.31.13:4629 , piece_id=GIXFWJNOMXONQCQIONLZNHZUKB65JL6VVPJ5CNWKUONQW25VRYMQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.554466571 (513h0m33.425549606s) [noise=true , address=145.79.227.15:4875 , piece_id=RSURGRNZPY3KNRXPYBRRF2AKOW4CW4WVBDEJGH5ZFAC5KJRUCB4A, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.554784676 (513h0m33.425867712s) [noise=true , address=77.90.31.10:4519 , piece_id=7ZQTTHLJCKSN55EJNCX3TD4YUSIE5EI2DOUKT2RFWOT4UMEMOTKA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.555093044 (513h0m33.426176089s) [noise=true , address=45.91.116.5:4299 , piece_id=M7FXCTF3JOEBEMMAMH5GP3HLW3OLXRD26HQLJOC2PVQT7VZUECPA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:35.261344284 (513h0m35.132427329s) [upload=450154]: Next returned nil segment Dec 27 00:29:35.261359944 (513h0m35.132442979s) [upload=450154]: Waiting for error group managing segments... Dec 27 00:29:35.553544942 (513h0m35.424627977s) [noise=true , address=145.79.227.6:4258 , piece_id=2Y3M2TY37O72IJM7GRAUPACI5FG2OIIYPFBPIHSB3KGIOSFZQXXQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.558212230 (513h0m35.429295275s) [noise=true , address=145.79.227.2:4113 , piece_id=KAKUAAKICZDRBTJRWYMZ2WFF623OBFYT65J6ESSSHSCKTYIK32MQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.560005734 (513h0m35.431088779s) [noise=true , address=145.79.227.37:4567 , piece_id=DPCYHB5ONQ7DGYFDQENSRILA7AAFDVO2OVKRT6X32VSOJAKQ5CJQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.560447175 (513h0m35.43153021s) [noise=true , address=145.79.227.15:4875 , piece_id=RSURGRNZPY3KNRXPYBRRF2AKOW4CW4WVBDEJGH5ZFAC5KJRUCB4A, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.560800788 (513h0m35.431883823s) [noise=true , address=145.79.227.51:4411 , piece_id=XPETBF2G6MVQ3VMGTX6ZG2QFS5555XKPSGCUXFQ57RG65UWH76KA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.583437762 (513h0m35.454520797s) [noise=true , address=145.79.227.47:4266 , piece_id=FAAG3Z5EBOMWKISWHUHFSZX2KTLVTIGEZRBEXWQ4FVQDKTWURDAQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.743623151 (513h0m35.614706196s) [noise=true , address=77.90.31.18:4808 , piece_id=C7NBOPC5GZ53KSG5U3MJW5ARCQPVK7XHYDBQNA642KNP4HGQF2HA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.754063620 (513h0m35.625146665s) [noise=true , address=77.90.31.13:4629 , piece_id=GIXFWJNOMXONQCQIONLZNHZUKB65JL6VVPJ5CNWKUONQW25VRYMQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.902109343 (513h0m35.773192388s) [noise=true , address=45.91.116.6:4974 , piece_id=2P4WX7V5SKN42QYPGLPFLN26IC5REVXSQ7RGZPGDPUH5VWV2Y5CA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.903974245 (513h0m35.77505728s) [noise=true , address=45.91.116.13:4155 , piece_id=LKJZYTNJZPUUOG3OODPM6TGMYY4WTLFAOGGC2SBCCIHG35WBTPBQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.904311547 (513h0m35.775394582s) [noise=true , address=45.91.116.10:4081 , piece_id=LZXI3WT4WFQYWG244GO7N7X6BAUU7QKKMIRHEDMJUCLGVDPIAGBQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.905333052 (513h0m35.776416087s) [noise=true , address=45.91.116.9:4264 , piece_id=IVBP4NTK4VY4EG7R3AKGFU662OO54I46XOFYUKCB6EN2SXSJDRSQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:36.168150111 (513h0m36.039233156s) [noise=true , address=45.91.116.5:4299 , piece_id=M7FXCTF3JOEBEMMAMH5GP3HLW3OLXRD26HQLJOC2PVQT7VZUECPA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:36.168171072 (513h0m36.039254107s) [seg_pos={0 0}, upload=450154]: Segment reached optimal threshold of 13 pieces. Dec 27 00:29:36.168343159 (513h0m36.039426204s) [noise=true , address=77.90.31.12:4589 , piece_id=HBJAIDCRKDGXVGBXIOWLWLFQ5UHOJKBYOBKWXNCZTCPFSJ6SXSNA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:49956->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:49956->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49956->77.90.31.12:4589: use of closed network connection Dec 27 00:29:36.168350814 (513h0m36.039433849s) [noise=true , address=77.90.31.12:4589 , piece_id=HBJAIDCRKDGXVGBXIOWLWLFQ5UHOJKBYOBKWXNCZTCPFSJ6SXSNA, seg_pos={0 0}, upload=450154]: Long tail cancellation due to optimal threshold Dec 27 00:29:36.168508604 (513h0m36.039591639s) [noise=true , address=45.91.116.14:4192 , piece_id=HBJ4PCTJABCHSHL53PUQWFEINL4BRTYBO3AGC544ZCMSLW6325FQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; write tcp 188.166.225.170:53916->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:53916->45.91.116.14:4192: use of closed network connection; EOF Dec 27 00:29:36.168524434 (513h0m36.039607469s) [noise=true , address=45.91.116.14:4192 , piece_id=HBJ4PCTJABCHSHL53PUQWFEINL4BRTYBO3AGC544ZCMSLW6325FQ, seg_pos={0 0}, upload=450154]: Long tail cancellation due to optimal threshold Dec 27 00:29:36.168617320 (513h0m36.039700356s) [noise=true , address=77.90.31.10:4519 , piece_id=7ZQTTHLJCKSN55EJNCX3TD4YUSIE5EI2DOUKT2RFWOT4UMEMOTKA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 00:29:36.168624264 (513h0m36.0397073s) [noise=true , address=77.90.31.10:4519 , piece_id=7ZQTTHLJCKSN55EJNCX3TD4YUSIE5EI2DOUKT2RFWOT4UMEMOTKA, seg_pos={0 0}, upload=450154]: Long tail cancellation due to optimal threshold Dec 27 00:29:36.168640325 (513h0m36.03972336s) [seg_pos={0 0}, upload=450154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:29:36.168731549 (513h0m36.039814584s) [upload=450154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:29:36.168737630 (513h0m36.039820665s) [upload=450154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:29:36.458582313 (513h0m36.329665348s) [upload=450154]: Done uploading segments... Dec 27 00:29:36.458601128 (513h0m36.329684164s) [upload=450154]: Upload finished. err: <nil> Duration: 6.000830007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.XX...X.........X.........X.......X.XX.XX.XXXX..X.XXX.X.X] duration: 30.004715973s trace: https://jaeger.dev.storj.io/trace/4eb1a943ce14edee FAIL (2025-12-27 00:30:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: LZXI3WT4WFQYWG244GO7N7X6BAUU7QKKMIRHEDMJUCLGVDPIAGBQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: GIXFWJNOMXONQCQIONLZNHZUKB65JL6VVPJ5CNWKUONQW25VRYMQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: M7FXCTF3JOEBEMMAMH5GP3HLW3OLXRD26HQLJOC2PVQT7VZUECPA) context deadline exceeded ======LOGS====== 29 (513h0m33.172672684s) [upload=450154]: Done starting upload Dec 27 00:29:33.301683027 (513h0m33.172766082s) [upload=450154]: Uploading segments... Dec 27 00:29:33.301943613 (513h0m33.173026668s) [upload=450154]: Got next segment. Inline: false Dec 27 00:29:33.302010490 (513h0m33.173093555s) [upload=450154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:29:33.302016772 (513h0m33.173099807s) [upload=450154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:29:33.550475352 (513h0m33.421558387s) [seg_pos={0 0}, upload=450154]: Begin upload segment... Dec 27 00:29:33.550519195 (513h0m33.42160223s) [seg_pos={0 0}, upload=450154]: Done begin upload segment. Dec 27 00:29:33.550544744 (513h0m33.421627779s) [noise=true , address=145.79.227.6:4258 , piece_id=2Y3M2TY37O72IJM7GRAUPACI5FG2OIIYPFBPIHSB3KGIOSFZQXXQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.550611070 (513h0m33.421694105s) [noise=true , address=145.79.227.2:4113 , piece_id=KAKUAAKICZDRBTJRWYMZ2WFF623OBFYT65J6ESSSHSCKTYIK32MQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.551163252 (513h0m33.422246287s) [noise=true , address=145.79.227.47:4266 , piece_id=FAAG3Z5EBOMWKISWHUHFSZX2KTLVTIGEZRBEXWQ4FVQDKTWURDAQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.551526693 (513h0m33.422609728s) [noise=true , address=77.90.31.12:4589 , piece_id=HBJAIDCRKDGXVGBXIOWLWLFQ5UHOJKBYOBKWXNCZTCPFSJ6SXSNA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.551970970 (513h0m33.423054005s) [noise=true , address=145.79.227.51:4411 , piece_id=XPETBF2G6MVQ3VMGTX6ZG2QFS5555XKPSGCUXFQ57RG65UWH76KA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.552216928 (513h0m33.423299963s) [noise=true , address=45.91.116.9:4264 , piece_id=IVBP4NTK4VY4EG7R3AKGFU662OO54I46XOFYUKCB6EN2SXSJDRSQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.552513452 (513h0m33.423596487s) [noise=true , address=77.90.31.18:4808 , piece_id=C7NBOPC5GZ53KSG5U3MJW5ARCQPVK7XHYDBQNA642KNP4HGQF2HA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.552769530 (513h0m33.423852565s) [noise=true , address=45.91.116.14:4192 , piece_id=HBJ4PCTJABCHSHL53PUQWFEINL4BRTYBO3AGC544ZCMSLW6325FQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553043312 (513h0m33.424126357s) [noise=true , address=45.91.116.6:4974 , piece_id=2P4WX7V5SKN42QYPGLPFLN26IC5REVXSQ7RGZPGDPUH5VWV2Y5CA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553320239 (513h0m33.424403284s) [noise=true , address=145.79.227.37:4567 , piece_id=DPCYHB5ONQ7DGYFDQENSRILA7AAFDVO2OVKRT6X32VSOJAKQ5CJQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553591014 (513h0m33.424674059s) [noise=true , address=45.91.116.13:4155 , piece_id=LKJZYTNJZPUUOG3OODPM6TGMYY4WTLFAOGGC2SBCCIHG35WBTPBQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.553891426 (513h0m33.424974461s) [noise=true , address=45.91.116.10:4081 , piece_id=LZXI3WT4WFQYWG244GO7N7X6BAUU7QKKMIRHEDMJUCLGVDPIAGBQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.554070657 (513h0m33.425153692s) [noise=true , address=77.90.31.13:4629 , piece_id=GIXFWJNOMXONQCQIONLZNHZUKB65JL6VVPJ5CNWKUONQW25VRYMQ, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.554466571 (513h0m33.425549606s) [noise=true , address=145.79.227.15:4875 , piece_id=RSURGRNZPY3KNRXPYBRRF2AKOW4CW4WVBDEJGH5ZFAC5KJRUCB4A, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.554784676 (513h0m33.425867712s) [noise=true , address=77.90.31.10:4519 , piece_id=7ZQTTHLJCKSN55EJNCX3TD4YUSIE5EI2DOUKT2RFWOT4UMEMOTKA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:33.555093044 (513h0m33.426176089s) [noise=true , address=45.91.116.5:4299 , piece_id=M7FXCTF3JOEBEMMAMH5GP3HLW3OLXRD26HQLJOC2PVQT7VZUECPA, seg_pos={0 0}, upload=450154]: Uploading piece... Dec 27 00:29:35.261344284 (513h0m35.132427329s) [upload=450154]: Next returned nil segment Dec 27 00:29:35.261359944 (513h0m35.132442979s) [upload=450154]: Waiting for error group managing segments... Dec 27 00:29:35.553544942 (513h0m35.424627977s) [noise=true , address=145.79.227.6:4258 , piece_id=2Y3M2TY37O72IJM7GRAUPACI5FG2OIIYPFBPIHSB3KGIOSFZQXXQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.558212230 (513h0m35.429295275s) [noise=true , address=145.79.227.2:4113 , piece_id=KAKUAAKICZDRBTJRWYMZ2WFF623OBFYT65J6ESSSHSCKTYIK32MQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.560005734 (513h0m35.431088779s) [noise=true , address=145.79.227.37:4567 , piece_id=DPCYHB5ONQ7DGYFDQENSRILA7AAFDVO2OVKRT6X32VSOJAKQ5CJQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.560447175 (513h0m35.43153021s) [noise=true , address=145.79.227.15:4875 , piece_id=RSURGRNZPY3KNRXPYBRRF2AKOW4CW4WVBDEJGH5ZFAC5KJRUCB4A, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.560800788 (513h0m35.431883823s) [noise=true , address=145.79.227.51:4411 , piece_id=XPETBF2G6MVQ3VMGTX6ZG2QFS5555XKPSGCUXFQ57RG65UWH76KA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.583437762 (513h0m35.454520797s) [noise=true , address=145.79.227.47:4266 , piece_id=FAAG3Z5EBOMWKISWHUHFSZX2KTLVTIGEZRBEXWQ4FVQDKTWURDAQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.743623151 (513h0m35.614706196s) [noise=true , address=77.90.31.18:4808 , piece_id=C7NBOPC5GZ53KSG5U3MJW5ARCQPVK7XHYDBQNA642KNP4HGQF2HA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.754063620 (513h0m35.625146665s) [noise=true , address=77.90.31.13:4629 , piece_id=GIXFWJNOMXONQCQIONLZNHZUKB65JL6VVPJ5CNWKUONQW25VRYMQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.902109343 (513h0m35.773192388s) [noise=true , address=45.91.116.6:4974 , piece_id=2P4WX7V5SKN42QYPGLPFLN26IC5REVXSQ7RGZPGDPUH5VWV2Y5CA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.903974245 (513h0m35.77505728s) [noise=true , address=45.91.116.13:4155 , piece_id=LKJZYTNJZPUUOG3OODPM6TGMYY4WTLFAOGGC2SBCCIHG35WBTPBQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.904311547 (513h0m35.775394582s) [noise=true , address=45.91.116.10:4081 , piece_id=LZXI3WT4WFQYWG244GO7N7X6BAUU7QKKMIRHEDMJUCLGVDPIAGBQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:35.905333052 (513h0m35.776416087s) [noise=true , address=45.91.116.9:4264 , piece_id=IVBP4NTK4VY4EG7R3AKGFU662OO54I46XOFYUKCB6EN2SXSJDRSQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:36.168150111 (513h0m36.039233156s) [noise=true , address=45.91.116.5:4299 , piece_id=M7FXCTF3JOEBEMMAMH5GP3HLW3OLXRD26HQLJOC2PVQT7VZUECPA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: <nil> Dec 27 00:29:36.168171072 (513h0m36.039254107s) [seg_pos={0 0}, upload=450154]: Segment reached optimal threshold of 13 pieces. Dec 27 00:29:36.168343159 (513h0m36.039426204s) [noise=true , address=77.90.31.12:4589 , piece_id=HBJAIDCRKDGXVGBXIOWLWLFQ5UHOJKBYOBKWXNCZTCPFSJ6SXSNA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:49956->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:49956->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49956->77.90.31.12:4589: use of closed network connection Dec 27 00:29:36.168350814 (513h0m36.039433849s) [noise=true , address=77.90.31.12:4589 , piece_id=HBJAIDCRKDGXVGBXIOWLWLFQ5UHOJKBYOBKWXNCZTCPFSJ6SXSNA, seg_pos={0 0}, upload=450154]: Long tail cancellation due to optimal threshold Dec 27 00:29:36.168508604 (513h0m36.039591639s) [noise=true , address=45.91.116.14:4192 , piece_id=HBJ4PCTJABCHSHL53PUQWFEINL4BRTYBO3AGC544ZCMSLW6325FQ, seg_pos={0 0}, upload=450154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; write tcp 188.166.225.170:53916->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:53916->45.91.116.14:4192: use of closed network connection; EOF Dec 27 00:29:36.168524434 (513h0m36.039607469s) [noise=true , address=45.91.116.14:4192 , piece_id=HBJ4PCTJABCHSHL53PUQWFEINL4BRTYBO3AGC544ZCMSLW6325FQ, seg_pos={0 0}, upload=450154]: Long tail cancellation due to optimal threshold Dec 27 00:29:36.168617320 (513h0m36.039700356s) [noise=true , address=77.90.31.10:4519 , piece_id=7ZQTTHLJCKSN55EJNCX3TD4YUSIE5EI2DOUKT2RFWOT4UMEMOTKA, seg_pos={0 0}, upload=450154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 00:29:36.168624264 (513h0m36.0397073s) [noise=true , address=77.90.31.10:4519 , piece_id=7ZQTTHLJCKSN55EJNCX3TD4YUSIE5EI2DOUKT2RFWOT4UMEMOTKA, seg_pos={0 0}, upload=450154]: Long tail cancellation due to optimal threshold Dec 27 00:29:36.168640325 (513h0m36.03972336s) [seg_pos={0 0}, upload=450154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:29:36.168731549 (513h0m36.039814584s) [upload=450154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:29:36.168737630 (513h0m36.039820665s) [upload=450154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:29:36.458582313 (513h0m36.329665348s) [upload=450154]: Done uploading segments... Dec 27 00:29:36.458601128 (513h0m36.329684164s) [upload=450154]: Upload finished. err: <nil> Duration: 30.004742183s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X.XX...X.........X.........X.......X.XX.XX.XXXX..X.XXX.X.] duration: 18.488738969s trace: https://jaeger.dev.storj.io/trace/2e60356ae2652646 FAIL (2025-12-27 00:28:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m50.966141007s) [upload=450133]: Done starting upload Dec 27 00:27:51.095092308 (512h58m50.966175343s) [upload=450133]: Uploading segments... Dec 27 00:27:51.095224179 (512h58m50.966307214s) [upload=450133]: Got next segment. Inline: false Dec 27 00:27:51.095248305 (512h58m50.96633134s) [upload=450133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:27:51.095252383 (512h58m50.966335418s) [upload=450133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:27:51.336821223 (512h58m51.207904268s) [seg_pos={0 0}, upload=450133]: Begin upload segment... Dec 27 00:27:51.336873433 (512h58m51.207956468s) [seg_pos={0 0}, upload=450133]: Done begin upload segment. Dec 27 00:27:51.336905895 (512h58m51.20798894s) [noise=true , address=145.79.227.13:4803 , piece_id=CTNG3FH3WDNLRQV2UJCLZKRMEH24QX2LKFWE3SHSUF3A6SNYDXSA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.336938497 (512h58m51.208021532s) [noise=true , address=45.91.116.7:4011 , piece_id=COX5C6AGER3EPUK5NEPHJCWHJMDGREXH74WEM7OMYI36GSVB3S3Q, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.337620024 (512h58m51.208703069s) [noise=true , address=145.79.227.11:4729 , piece_id=CIFVPWDVD53LJQSKAS64MKN2PF2ORIEPTSE2BDRR7TCEA7JQN4BA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.338031408 (512h58m51.209114443s) [noise=true , address=45.91.116.6:4973 , piece_id=BWEGWPQPMSTEDLFWRH6DBMO5UXUARSJDMJXHORYU3IRETYBW7MOA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.338425398 (512h58m51.209508443s) [noise=true , address=145.79.227.12:4765 , piece_id=45W7OFXTZLRUBF723Z7NH7DMOIGLVCFMRASSTB3EIB6MZQJAMTDA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.338789411 (512h58m51.209872446s) [noise=true , address=145.79.227.36:4519 , piece_id=P3XCDJTAG7WCK6CWH3KPC2T3664AUWRWOCYQMSPURLSPGLWSIRVQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.339182960 (512h58m51.210265995s) [noise=true , address=145.79.227.42:4121 , piece_id=WGWPZUYOQBQLRUEWIGJL4FUHVUFS7AS4PKKPCIXRHQKBTRDF7H7A, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.339531845 (512h58m51.21061489s) [noise=true , address=77.90.31.21:4878 , piece_id=IRAG76UIG5LSYZUYDYSNDFARM5LATELLNLTN64KBZEUAGKHQ5V5Q, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.339890688 (512h58m51.210973723s) [noise=true , address=145.79.227.46:4603 , piece_id=4JO44OR7TMIDEMPYOXOONYFVTCXUVWMIVGDTYNUWLFKRC4QXSOBQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.340241956 (512h58m51.211324992s) [noise=true , address=45.91.116.8:4046 , piece_id=35XULOSJFO7LXHSNFK3HT5L5HB4OWWILO6R3RCIH6IH3NTCJUZDA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.340557798 (512h58m51.211640833s) [noise=true , address=145.79.227.45:4229 , piece_id=K65TXYSBVTUEX3HG33RKHOULWWF734NRDW3FHN6XRXOLWBSQFKZQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.340947661 (512h58m51.212030706s) [noise=true , address=77.90.31.13:4627 , piece_id=K7OEZX7JDCHRDQHOPIOEQOGTPSOU2UPWRDATX7NI374CELYA45MQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.341230009 (512h58m51.212313044s) [noise=true , address=45.91.116.12:4226 , piece_id=ZXDESEJDPLZFAZJPEEKEKC6YOWNGA7TXBRMORKFL7DP4QWIIVGDQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.341995777 (512h58m51.213078812s) [noise=true , address=45.91.116.5:4298 , piece_id=BWFVBG5BTEBVFGLP63KLMOWRM4V3KPBFMYGJOTGMGDAAZUKRHFKQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.342416327 (512h58m51.213499362s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLAD747RML7NUJXTMSVVAYKYAGRZAHN7IURV7BD3ENJ3HWEN4BQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.342759962 (512h58m51.213842997s) [noise=true , address=77.90.31.11:4556 , piece_id=63ZZPZPCEXAT6D6CLARKBVUEEUXV2FR3O6KEA547NZ6VD7FHCHZA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:53.056010430 (512h58m52.927093475s) [upload=450133]: Next returned nil segment Dec 27 00:27:53.056024867 (512h58m52.927107912s) [upload=450133]: Waiting for error group managing segments... Dec 27 00:27:53.330344769 (512h58m53.201427824s) [noise=true , address=145.79.227.12:4765 , piece_id=45W7OFXTZLRUBF723Z7NH7DMOIGLVCFMRASSTB3EIB6MZQJAMTDA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.332791529 (512h58m53.203874574s) [noise=true , address=145.79.227.11:4729 , piece_id=CIFVPWDVD53LJQSKAS64MKN2PF2ORIEPTSE2BDRR7TCEA7JQN4BA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.349249506 (512h58m53.220332551s) [noise=true , address=145.79.227.42:4121 , piece_id=WGWPZUYOQBQLRUEWIGJL4FUHVUFS7AS4PKKPCIXRHQKBTRDF7H7A, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.350754812 (512h58m53.221837857s) [noise=true , address=145.79.227.45:4229 , piece_id=K65TXYSBVTUEX3HG33RKHOULWWF734NRDW3FHN6XRXOLWBSQFKZQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.351708517 (512h58m53.222791563s) [noise=true , address=145.79.227.36:4519 , piece_id=P3XCDJTAG7WCK6CWH3KPC2T3664AUWRWOCYQMSPURLSPGLWSIRVQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.361591386 (512h58m53.232674431s) [noise=true , address=145.79.227.46:4603 , piece_id=4JO44OR7TMIDEMPYOXOONYFVTCXUVWMIVGDTYNUWLFKRC4QXSOBQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.381886720 (512h58m53.252969765s) [noise=true , address=145.79.227.13:4803 , piece_id=CTNG3FH3WDNLRQV2UJCLZKRMEH24QX2LKFWE3SHSUF3A6SNYDXSA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.528012939 (512h58m53.399095994s) [noise=true , address=77.90.31.21:4878 , piece_id=IRAG76UIG5LSYZUYDYSNDFARM5LATELLNLTN64KBZEUAGKHQ5V5Q, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.611275740 (512h58m53.482358775s) [noise=true , address=45.91.116.6:4973 , piece_id=BWEGWPQPMSTEDLFWRH6DBMO5UXUARSJDMJXHORYU3IRETYBW7MOA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.643915699 (512h58m53.514998734s) [noise=true , address=77.90.31.11:4556 , piece_id=63ZZPZPCEXAT6D6CLARKBVUEEUXV2FR3O6KEA547NZ6VD7FHCHZA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.692898517 (512h58m53.563981562s) [noise=true , address=45.91.116.12:4226 , piece_id=ZXDESEJDPLZFAZJPEEKEKC6YOWNGA7TXBRMORKFL7DP4QWIIVGDQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.698529950 (512h58m53.569612995s) [noise=true , address=45.91.116.5:4298 , piece_id=BWFVBG5BTEBVFGLP63KLMOWRM4V3KPBFMYGJOTGMGDAAZUKRHFKQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.753334143 (512h58m53.624417178s) [noise=true , address=77.90.31.13:4627 , piece_id=K7OEZX7JDCHRDQHOPIOEQOGTPSOU2UPWRDATX7NI374CELYA45MQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.753352407 (512h58m53.624435452s) [seg_pos={0 0}, upload=450133]: Segment reached optimal threshold of 13 pieces. Dec 27 00:27:53.753528953 (512h58m53.624611988s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLAD747RML7NUJXTMSVVAYKYAGRZAHN7IURV7BD3ENJ3HWEN4BQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:54968->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:54968->145.79.227.41:4085: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54968->145.79.227.41:4085: use of closed network connection Dec 27 00:27:53.753535366 (512h58m53.624618401s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLAD747RML7NUJXTMSVVAYKYAGRZAHN7IURV7BD3ENJ3HWEN4BQ, seg_pos={0 0}, upload=450133]: Long tail cancellation due to optimal threshold Dec 27 00:27:53.753824506 (512h58m53.624907541s) [noise=true , address=45.91.116.7:4011 , piece_id=COX5C6AGER3EPUK5NEPHJCWHJMDGREXH74WEM7OMYI36GSVB3S3Q, seg_pos={0 0}, upload=450133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 00:27:53.753830838 (512h58m53.624913873s) [noise=true , address=45.91.116.7:4011 , piece_id=COX5C6AGER3EPUK5NEPHJCWHJMDGREXH74WEM7OMYI36GSVB3S3Q, seg_pos={0 0}, upload=450133]: Long tail cancellation due to optimal threshold Dec 27 00:27:53.754121581 (512h58m53.625204626s) [noise=true , address=45.91.116.8:4046 , piece_id=35XULOSJFO7LXHSNFK3HT5L5HB4OWWILO6R3RCIH6IH3NTCJUZDA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: read tcp 188.166.225.170:51930->45.91.116.8:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 00:27:53.754220660 (512h58m53.625303725s) [noise=true , address=45.91.116.8:4046 , piece_id=35XULOSJFO7LXHSNFK3HT5L5HB4OWWILO6R3RCIH6IH3NTCJUZDA, seg_pos={0 0}, upload=450133]: Long tail cancellation due to optimal threshold Dec 27 00:27:53.754241450 (512h58m53.625324495s) [seg_pos={0 0}, upload=450133]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:27:53.754389642 (512h58m53.625472677s) [upload=450133]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:27:53.754395643 (512h58m53.625478688s) [upload=450133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:27:54.046597816 (512h58m53.917681091s) [upload=450133]: Done uploading segments... Dec 27 00:27:54.046620889 (512h58m53.917703934s) [upload=450133]: Upload finished. err: <nil> Duration: 18.488767654s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 252.26641ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:28:33 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 252.291389ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX..X.XX...X.........X.........X.......X.XX.XX.XXXX..X.XXX.X] duration: 30.002857868s trace: https://jaeger.dev.storj.io/trace/2e60356ae2652646 FAIL (2025-12-27 00:28:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m50.966141007s) [upload=450133]: Done starting upload Dec 27 00:27:51.095092308 (512h58m50.966175343s) [upload=450133]: Uploading segments... Dec 27 00:27:51.095224179 (512h58m50.966307214s) [upload=450133]: Got next segment. Inline: false Dec 27 00:27:51.095248305 (512h58m50.96633134s) [upload=450133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:27:51.095252383 (512h58m50.966335418s) [upload=450133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:27:51.336821223 (512h58m51.207904268s) [seg_pos={0 0}, upload=450133]: Begin upload segment... Dec 27 00:27:51.336873433 (512h58m51.207956468s) [seg_pos={0 0}, upload=450133]: Done begin upload segment. Dec 27 00:27:51.336905895 (512h58m51.20798894s) [noise=true , address=145.79.227.13:4803 , piece_id=CTNG3FH3WDNLRQV2UJCLZKRMEH24QX2LKFWE3SHSUF3A6SNYDXSA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.336938497 (512h58m51.208021532s) [noise=true , address=45.91.116.7:4011 , piece_id=COX5C6AGER3EPUK5NEPHJCWHJMDGREXH74WEM7OMYI36GSVB3S3Q, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.337620024 (512h58m51.208703069s) [noise=true , address=145.79.227.11:4729 , piece_id=CIFVPWDVD53LJQSKAS64MKN2PF2ORIEPTSE2BDRR7TCEA7JQN4BA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.338031408 (512h58m51.209114443s) [noise=true , address=45.91.116.6:4973 , piece_id=BWEGWPQPMSTEDLFWRH6DBMO5UXUARSJDMJXHORYU3IRETYBW7MOA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.338425398 (512h58m51.209508443s) [noise=true , address=145.79.227.12:4765 , piece_id=45W7OFXTZLRUBF723Z7NH7DMOIGLVCFMRASSTB3EIB6MZQJAMTDA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.338789411 (512h58m51.209872446s) [noise=true , address=145.79.227.36:4519 , piece_id=P3XCDJTAG7WCK6CWH3KPC2T3664AUWRWOCYQMSPURLSPGLWSIRVQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.339182960 (512h58m51.210265995s) [noise=true , address=145.79.227.42:4121 , piece_id=WGWPZUYOQBQLRUEWIGJL4FUHVUFS7AS4PKKPCIXRHQKBTRDF7H7A, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.339531845 (512h58m51.21061489s) [noise=true , address=77.90.31.21:4878 , piece_id=IRAG76UIG5LSYZUYDYSNDFARM5LATELLNLTN64KBZEUAGKHQ5V5Q, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.339890688 (512h58m51.210973723s) [noise=true , address=145.79.227.46:4603 , piece_id=4JO44OR7TMIDEMPYOXOONYFVTCXUVWMIVGDTYNUWLFKRC4QXSOBQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.340241956 (512h58m51.211324992s) [noise=true , address=45.91.116.8:4046 , piece_id=35XULOSJFO7LXHSNFK3HT5L5HB4OWWILO6R3RCIH6IH3NTCJUZDA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.340557798 (512h58m51.211640833s) [noise=true , address=145.79.227.45:4229 , piece_id=K65TXYSBVTUEX3HG33RKHOULWWF734NRDW3FHN6XRXOLWBSQFKZQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.340947661 (512h58m51.212030706s) [noise=true , address=77.90.31.13:4627 , piece_id=K7OEZX7JDCHRDQHOPIOEQOGTPSOU2UPWRDATX7NI374CELYA45MQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.341230009 (512h58m51.212313044s) [noise=true , address=45.91.116.12:4226 , piece_id=ZXDESEJDPLZFAZJPEEKEKC6YOWNGA7TXBRMORKFL7DP4QWIIVGDQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.341995777 (512h58m51.213078812s) [noise=true , address=45.91.116.5:4298 , piece_id=BWFVBG5BTEBVFGLP63KLMOWRM4V3KPBFMYGJOTGMGDAAZUKRHFKQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.342416327 (512h58m51.213499362s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLAD747RML7NUJXTMSVVAYKYAGRZAHN7IURV7BD3ENJ3HWEN4BQ, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:51.342759962 (512h58m51.213842997s) [noise=true , address=77.90.31.11:4556 , piece_id=63ZZPZPCEXAT6D6CLARKBVUEEUXV2FR3O6KEA547NZ6VD7FHCHZA, seg_pos={0 0}, upload=450133]: Uploading piece... Dec 27 00:27:53.056010430 (512h58m52.927093475s) [upload=450133]: Next returned nil segment Dec 27 00:27:53.056024867 (512h58m52.927107912s) [upload=450133]: Waiting for error group managing segments... Dec 27 00:27:53.330344769 (512h58m53.201427824s) [noise=true , address=145.79.227.12:4765 , piece_id=45W7OFXTZLRUBF723Z7NH7DMOIGLVCFMRASSTB3EIB6MZQJAMTDA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.332791529 (512h58m53.203874574s) [noise=true , address=145.79.227.11:4729 , piece_id=CIFVPWDVD53LJQSKAS64MKN2PF2ORIEPTSE2BDRR7TCEA7JQN4BA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.349249506 (512h58m53.220332551s) [noise=true , address=145.79.227.42:4121 , piece_id=WGWPZUYOQBQLRUEWIGJL4FUHVUFS7AS4PKKPCIXRHQKBTRDF7H7A, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.350754812 (512h58m53.221837857s) [noise=true , address=145.79.227.45:4229 , piece_id=K65TXYSBVTUEX3HG33RKHOULWWF734NRDW3FHN6XRXOLWBSQFKZQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.351708517 (512h58m53.222791563s) [noise=true , address=145.79.227.36:4519 , piece_id=P3XCDJTAG7WCK6CWH3KPC2T3664AUWRWOCYQMSPURLSPGLWSIRVQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.361591386 (512h58m53.232674431s) [noise=true , address=145.79.227.46:4603 , piece_id=4JO44OR7TMIDEMPYOXOONYFVTCXUVWMIVGDTYNUWLFKRC4QXSOBQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.381886720 (512h58m53.252969765s) [noise=true , address=145.79.227.13:4803 , piece_id=CTNG3FH3WDNLRQV2UJCLZKRMEH24QX2LKFWE3SHSUF3A6SNYDXSA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.528012939 (512h58m53.399095994s) [noise=true , address=77.90.31.21:4878 , piece_id=IRAG76UIG5LSYZUYDYSNDFARM5LATELLNLTN64KBZEUAGKHQ5V5Q, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.611275740 (512h58m53.482358775s) [noise=true , address=45.91.116.6:4973 , piece_id=BWEGWPQPMSTEDLFWRH6DBMO5UXUARSJDMJXHORYU3IRETYBW7MOA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.643915699 (512h58m53.514998734s) [noise=true , address=77.90.31.11:4556 , piece_id=63ZZPZPCEXAT6D6CLARKBVUEEUXV2FR3O6KEA547NZ6VD7FHCHZA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.692898517 (512h58m53.563981562s) [noise=true , address=45.91.116.12:4226 , piece_id=ZXDESEJDPLZFAZJPEEKEKC6YOWNGA7TXBRMORKFL7DP4QWIIVGDQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.698529950 (512h58m53.569612995s) [noise=true , address=45.91.116.5:4298 , piece_id=BWFVBG5BTEBVFGLP63KLMOWRM4V3KPBFMYGJOTGMGDAAZUKRHFKQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.753334143 (512h58m53.624417178s) [noise=true , address=77.90.31.13:4627 , piece_id=K7OEZX7JDCHRDQHOPIOEQOGTPSOU2UPWRDATX7NI374CELYA45MQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: <nil> Dec 27 00:27:53.753352407 (512h58m53.624435452s) [seg_pos={0 0}, upload=450133]: Segment reached optimal threshold of 13 pieces. Dec 27 00:27:53.753528953 (512h58m53.624611988s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLAD747RML7NUJXTMSVVAYKYAGRZAHN7IURV7BD3ENJ3HWEN4BQ, seg_pos={0 0}, upload=450133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:54968->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:54968->145.79.227.41:4085: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54968->145.79.227.41:4085: use of closed network connection Dec 27 00:27:53.753535366 (512h58m53.624618401s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLAD747RML7NUJXTMSVVAYKYAGRZAHN7IURV7BD3ENJ3HWEN4BQ, seg_pos={0 0}, upload=450133]: Long tail cancellation due to optimal threshold Dec 27 00:27:53.753824506 (512h58m53.624907541s) [noise=true , address=45.91.116.7:4011 , piece_id=COX5C6AGER3EPUK5NEPHJCWHJMDGREXH74WEM7OMYI36GSVB3S3Q, seg_pos={0 0}, upload=450133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 00:27:53.753830838 (512h58m53.624913873s) [noise=true , address=45.91.116.7:4011 , piece_id=COX5C6AGER3EPUK5NEPHJCWHJMDGREXH74WEM7OMYI36GSVB3S3Q, seg_pos={0 0}, upload=450133]: Long tail cancellation due to optimal threshold Dec 27 00:27:53.754121581 (512h58m53.625204626s) [noise=true , address=45.91.116.8:4046 , piece_id=35XULOSJFO7LXHSNFK3HT5L5HB4OWWILO6R3RCIH6IH3NTCJUZDA, seg_pos={0 0}, upload=450133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: read tcp 188.166.225.170:51930->45.91.116.8:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 00:27:53.754220660 (512h58m53.625303725s) [noise=true , address=45.91.116.8:4046 , piece_id=35XULOSJFO7LXHSNFK3HT5L5HB4OWWILO6R3RCIH6IH3NTCJUZDA, seg_pos={0 0}, upload=450133]: Long tail cancellation due to optimal threshold Dec 27 00:27:53.754241450 (512h58m53.625324495s) [seg_pos={0 0}, upload=450133]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:27:53.754389642 (512h58m53.625472677s) [upload=450133]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:27:53.754395643 (512h58m53.625478688s) [upload=450133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:27:54.046597816 (512h58m53.917681091s) [upload=450133]: Done uploading segments... Dec 27 00:27:54.046620889 (512h58m53.917703934s) [upload=450133]: Upload finished. err: <nil> Duration: 30.002880502s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 259.403799ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:27:40 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 259.427757ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX..X.XX...X.........X.........X.......X.XX.XX.XXXX..X.XXX.] duration: 12.623038666s trace: https://jaeger.dev.storj.io/trace/31af01fb83ff03d8 FAIL (2025-12-27 00:27:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: ZW7X5OSJRVOH6SRIW4SE6IHZSOTAHGPEV65OAIXRJHJO23SLK2QA) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: RX2UH6EXJJKNQMMPHEG2ZOSR7NNYMTQ6OVR4S63RMCAKOYXJ5ZJA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: F3VILBFRIARIPRCI7FZ7T6UC3HF2OULRLY4B7LBNJKRIVV6ILYNQ) context deadline exceeded ======LOGS====== Dec 27 00:26:13.485868617 (512h57m13.356951652s) [upload=450111]: Starting upload Dec 27 00:26:13.485948909 (512h57m13.357031944s) [upload=450111]: Done starting upload Dec 27 00:26:13.486031827 (512h57m13.357114872s) [upload=450111]: Uploading segments... Dec 27 00:26:13.486229604 (512h57m13.357312639s) [upload=450111]: Got next segment. Inline: false Dec 27 00:26:13.486271273 (512h57m13.357354308s) [upload=450111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:26:13.486277125 (512h57m13.35736016s) [upload=450111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:26:13.730366795 (512h57m13.60144983s) [seg_pos={0 0}, upload=450111]: Begin upload segment... Dec 27 00:26:13.730415978 (512h57m13.601499013s) [seg_pos={0 0}, upload=450111]: Done begin upload segment. Dec 27 00:26:13.730443431 (512h57m13.601526476s) [noise=true , address=145.79.227.46:4601 , piece_id=IZ53F5AB6WVLSUAZZVCSKDW5GH6OHTPHVRSGM27OUJIHSAAK2BTA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.733730039 (512h57m13.604813085s) [noise=true , address=45.91.116.8:4046 , piece_id=ZW7X5OSJRVOH6SRIW4SE6IHZSOTAHGPEV65OAIXRJHJO23SLK2QA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.734361502 (512h57m13.605444547s) [noise=true , address=45.91.116.7:4009 , piece_id=NIIYPJLFDPQS6523FP2J3HMJRYFXPPGIQ76PSGDMCIMWVUZEDXZA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.735228472 (512h57m13.606311507s) [noise=true , address=45.91.116.13:4155 , piece_id=2VRBWI6GMSR45ZZCTTY6CJAUQCXFBRYRPRUSCFIKHCEFPP3QHQ4A, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.737941919 (512h57m13.609024964s) [noise=true , address=45.91.116.5:4297 , piece_id=KP2KL5UU2H2ASJNCBWHC5DD2UME5TLFDCKCRBMVHNP4QAMIVM4UA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.739424351 (512h57m13.610507396s) [noise=true , address=77.90.31.10:4517 , piece_id=RX2UH6EXJJKNQMMPHEG2ZOSR7NNYMTQ6OVR4S63RMCAKOYXJ5ZJA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.740935970 (512h57m13.612019005s) [noise=true , address=145.79.227.37:4567 , piece_id=M7TFXHUWQDFNVHKIMCF5KFIGMPYCLKPBHSOKONSPWMA7F67RTLBQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.743070324 (512h57m13.614153369s) [noise=true , address=77.90.31.18:4805 , piece_id=GMBXRNG42MCB2VETHCZHTWQKE5JJMLQ4FKZILXPXKOJCPYABTNVQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.743960358 (512h57m13.615043403s) [noise=true , address=145.79.227.42:4122 , piece_id=I2KS77ZLJTKDKQY2ADJW3LGPN3PSXSNR3ZDQNEDISPXITN2ZWKPQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.744410625 (512h57m13.61549367s) [noise=true , address=145.79.227.16:4911 , piece_id=WWR4HNZHSUAC5BUMODIRWKZJVS5XHGNY52HDXFGQ6I2WB76E2XIQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.745240746 (512h57m13.616323781s) [noise=true , address=45.91.116.10:4082 , piece_id=F3VILBFRIARIPRCI7FZ7T6UC3HF2OULRLY4B7LBNJKRIVV6ILYNQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.746879296 (512h57m13.617962341s) [noise=true , address=77.90.31.13:4627 , piece_id=BUEM2WTXD7F6H4ZJYCE5H5E7RXQHZBKYMEKT6FYJC2OAO6JBNZRA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.747140093 (512h57m13.618223128s) [noise=true , address=145.79.227.41:4087 , piece_id=QXFO4PEXDZKP7LNVA7VBJQT3JPKWG4HUDEZHW5EY2NMVUV4FJORQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.751890168 (512h57m13.622973213s) [noise=true , address=77.90.31.17:4773 , piece_id=IVXE3ZHEAF3JQO27NXEYP2EZHG3XXZRVXW3LXN5WNOH2SJY3KS2A, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.752023742 (512h57m13.623106787s) [noise=true , address=77.90.31.12:4590 , piece_id=CZCNJEG34EX7XIHMKE7YHABT7ITHDIYERGB6R2AL3CQVBZWZYV4Q, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.755018765 (512h57m13.62610181s) [noise=true , address=77.90.31.11:4553 , piece_id=GVARHA2VBANU4DJZ2KPF46D2PKETF7EWN4BF4MSSL2Q2FM4CV4AA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:15.457423130 (512h57m15.328506165s) [upload=450111]: Next returned nil segment Dec 27 00:26:15.457443779 (512h57m15.328526815s) [upload=450111]: Waiting for error group managing segments... Dec 27 00:26:15.758087324 (512h57m15.629170359s) [noise=true , address=145.79.227.42:4122 , piece_id=I2KS77ZLJTKDKQY2ADJW3LGPN3PSXSNR3ZDQNEDISPXITN2ZWKPQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:15.767345973 (512h57m15.638429008s) [noise=true , address=145.79.227.37:4567 , piece_id=M7TFXHUWQDFNVHKIMCF5KFIGMPYCLKPBHSOKONSPWMA7F67RTLBQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:15.769246331 (512h57m15.640329366s) [noise=true , address=145.79.227.46:4601 , piece_id=IZ53F5AB6WVLSUAZZVCSKDW5GH6OHTPHVRSGM27OUJIHSAAK2BTA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:15.777540344 (512h57m15.648623379s) [noise=true , address=145.79.227.16:4911 , piece_id=WWR4HNZHSUAC5BUMODIRWKZJVS5XHGNY52HDXFGQ6I2WB76E2XIQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:15.843997747 (512h57m15.715080782s) [noise=true , address=145.79.227.41:4087 , piece_id=QXFO4PEXDZKP7LNVA7VBJQT3JPKWG4HUDEZHW5EY2NMVUV4FJORQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:15.961524649 (512h57m15.832607684s) [noise=true , address=45.91.116.8:4046 , piece_id=ZW7X5OSJRVOH6SRIW4SE6IHZSOTAHGPEV65OAIXRJHJO23SLK2QA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.010426414 (512h57m15.881509459s) [noise=true , address=45.91.116.5:4297 , piece_id=KP2KL5UU2H2ASJNCBWHC5DD2UME5TLFDCKCRBMVHNP4QAMIVM4UA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.016228541 (512h57m15.887311576s) [noise=true , address=77.90.31.11:4553 , piece_id=GVARHA2VBANU4DJZ2KPF46D2PKETF7EWN4BF4MSSL2Q2FM4CV4AA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.021859053 (512h57m15.892942098s) [noise=true , address=77.90.31.18:4805 , piece_id=GMBXRNG42MCB2VETHCZHTWQKE5JJMLQ4FKZILXPXKOJCPYABTNVQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.038785783 (512h57m15.909868818s) [noise=true , address=45.91.116.13:4155 , piece_id=2VRBWI6GMSR45ZZCTTY6CJAUQCXFBRYRPRUSCFIKHCEFPP3QHQ4A, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.135394756 (512h57m16.006477791s) [noise=true , address=45.91.116.10:4082 , piece_id=F3VILBFRIARIPRCI7FZ7T6UC3HF2OULRLY4B7LBNJKRIVV6ILYNQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.167091479 (512h57m16.038174525s) [noise=true , address=77.90.31.13:4627 , piece_id=BUEM2WTXD7F6H4ZJYCE5H5E7RXQHZBKYMEKT6FYJC2OAO6JBNZRA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.266401165 (512h57m16.13748421s) [noise=true , address=77.90.31.10:4517 , piece_id=RX2UH6EXJJKNQMMPHEG2ZOSR7NNYMTQ6OVR4S63RMCAKOYXJ5ZJA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.266419109 (512h57m16.137502144s) [seg_pos={0 0}, upload=450111]: Segment reached optimal threshold of 13 pieces. Dec 27 00:26:16.266618118 (512h57m16.137701414s) [noise=true , address=77.90.31.17:4773 , piece_id=IVXE3ZHEAF3JQO27NXEYP2EZHG3XXZRVXW3LXN5WNOH2SJY3KS2A, seg_pos={0 0}, upload=450111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:38764->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:38764->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38764->77.90.31.17:4773: use of closed network connection Dec 27 00:26:16.266629960 (512h57m16.137713006s) [noise=true , address=77.90.31.17:4773 , piece_id=IVXE3ZHEAF3JQO27NXEYP2EZHG3XXZRVXW3LXN5WNOH2SJY3KS2A, seg_pos={0 0}, upload=450111]: Long tail cancellation due to optimal threshold Dec 27 00:26:16.266907339 (512h57m16.137990384s) [noise=true , address=77.90.31.12:4590 , piece_id=CZCNJEG34EX7XIHMKE7YHABT7ITHDIYERGB6R2AL3CQVBZWZYV4Q, seg_pos={0 0}, upload=450111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 00:26:16.266914172 (512h57m16.137997207s) [noise=true , address=77.90.31.12:4590 , piece_id=CZCNJEG34EX7XIHMKE7YHABT7ITHDIYERGB6R2AL3CQVBZWZYV4Q, seg_pos={0 0}, upload=450111]: Long tail cancellation due to optimal threshold Dec 27 00:26:16.266988603 (512h57m16.138071648s) [noise=true , address=45.91.116.7:4009 , piece_id=NIIYPJLFDPQS6523FP2J3HMJRYFXPPGIQ76PSGDMCIMWVUZEDXZA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 00:26:16.267005967 (512h57m16.138089002s) [noise=true , address=45.91.116.7:4009 , piece_id=NIIYPJLFDPQS6523FP2J3HMJRYFXPPGIQ76PSGDMCIMWVUZEDXZA, seg_pos={0 0}, upload=450111]: Long tail cancellation due to optimal threshold Dec 27 00:26:16.267054619 (512h57m16.138137654s) [seg_pos={0 0}, upload=450111]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:26:16.267163045 (512h57m16.13824608s) [upload=450111]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:26:16.267169428 (512h57m16.138252463s) [upload=450111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:26:16.556960039 (512h57m16.428043094s) [upload=450111]: Done uploading segments... Dec 27 00:26:16.556991860 (512h57m16.428074905s) [upload=450111]: Upload finished. err: <nil> Duration: 12.623097388s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 264.112898ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:27:07 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 264.133406ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXX..X.XX...X.........X.........X.......X.XX.XX.XXXX..X.XXX] duration: 30.003117246s trace: https://jaeger.dev.storj.io/trace/31af01fb83ff03d8 FAIL (2025-12-27 00:26:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: ZW7X5OSJRVOH6SRIW4SE6IHZSOTAHGPEV65OAIXRJHJO23SLK2QA) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: RX2UH6EXJJKNQMMPHEG2ZOSR7NNYMTQ6OVR4S63RMCAKOYXJ5ZJA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: F3VILBFRIARIPRCI7FZ7T6UC3HF2OULRLY4B7LBNJKRIVV6ILYNQ) context deadline exceeded ======LOGS====== Dec 27 00:26:13.485868617 (512h57m13.356951652s) [upload=450111]: Starting upload Dec 27 00:26:13.485948909 (512h57m13.357031944s) [upload=450111]: Done starting upload Dec 27 00:26:13.486031827 (512h57m13.357114872s) [upload=450111]: Uploading segments... Dec 27 00:26:13.486229604 (512h57m13.357312639s) [upload=450111]: Got next segment. Inline: false Dec 27 00:26:13.486271273 (512h57m13.357354308s) [upload=450111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:26:13.486277125 (512h57m13.35736016s) [upload=450111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:26:13.730366795 (512h57m13.60144983s) [seg_pos={0 0}, upload=450111]: Begin upload segment... Dec 27 00:26:13.730415978 (512h57m13.601499013s) [seg_pos={0 0}, upload=450111]: Done begin upload segment. Dec 27 00:26:13.730443431 (512h57m13.601526476s) [noise=true , address=145.79.227.46:4601 , piece_id=IZ53F5AB6WVLSUAZZVCSKDW5GH6OHTPHVRSGM27OUJIHSAAK2BTA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.733730039 (512h57m13.604813085s) [noise=true , address=45.91.116.8:4046 , piece_id=ZW7X5OSJRVOH6SRIW4SE6IHZSOTAHGPEV65OAIXRJHJO23SLK2QA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.734361502 (512h57m13.605444547s) [noise=true , address=45.91.116.7:4009 , piece_id=NIIYPJLFDPQS6523FP2J3HMJRYFXPPGIQ76PSGDMCIMWVUZEDXZA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.735228472 (512h57m13.606311507s) [noise=true , address=45.91.116.13:4155 , piece_id=2VRBWI6GMSR45ZZCTTY6CJAUQCXFBRYRPRUSCFIKHCEFPP3QHQ4A, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.737941919 (512h57m13.609024964s) [noise=true , address=45.91.116.5:4297 , piece_id=KP2KL5UU2H2ASJNCBWHC5DD2UME5TLFDCKCRBMVHNP4QAMIVM4UA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.739424351 (512h57m13.610507396s) [noise=true , address=77.90.31.10:4517 , piece_id=RX2UH6EXJJKNQMMPHEG2ZOSR7NNYMTQ6OVR4S63RMCAKOYXJ5ZJA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.740935970 (512h57m13.612019005s) [noise=true , address=145.79.227.37:4567 , piece_id=M7TFXHUWQDFNVHKIMCF5KFIGMPYCLKPBHSOKONSPWMA7F67RTLBQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.743070324 (512h57m13.614153369s) [noise=true , address=77.90.31.18:4805 , piece_id=GMBXRNG42MCB2VETHCZHTWQKE5JJMLQ4FKZILXPXKOJCPYABTNVQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.743960358 (512h57m13.615043403s) [noise=true , address=145.79.227.42:4122 , piece_id=I2KS77ZLJTKDKQY2ADJW3LGPN3PSXSNR3ZDQNEDISPXITN2ZWKPQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.744410625 (512h57m13.61549367s) [noise=true , address=145.79.227.16:4911 , piece_id=WWR4HNZHSUAC5BUMODIRWKZJVS5XHGNY52HDXFGQ6I2WB76E2XIQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.745240746 (512h57m13.616323781s) [noise=true , address=45.91.116.10:4082 , piece_id=F3VILBFRIARIPRCI7FZ7T6UC3HF2OULRLY4B7LBNJKRIVV6ILYNQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.746879296 (512h57m13.617962341s) [noise=true , address=77.90.31.13:4627 , piece_id=BUEM2WTXD7F6H4ZJYCE5H5E7RXQHZBKYMEKT6FYJC2OAO6JBNZRA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.747140093 (512h57m13.618223128s) [noise=true , address=145.79.227.41:4087 , piece_id=QXFO4PEXDZKP7LNVA7VBJQT3JPKWG4HUDEZHW5EY2NMVUV4FJORQ, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.751890168 (512h57m13.622973213s) [noise=true , address=77.90.31.17:4773 , piece_id=IVXE3ZHEAF3JQO27NXEYP2EZHG3XXZRVXW3LXN5WNOH2SJY3KS2A, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.752023742 (512h57m13.623106787s) [noise=true , address=77.90.31.12:4590 , piece_id=CZCNJEG34EX7XIHMKE7YHABT7ITHDIYERGB6R2AL3CQVBZWZYV4Q, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:13.755018765 (512h57m13.62610181s) [noise=true , address=77.90.31.11:4553 , piece_id=GVARHA2VBANU4DJZ2KPF46D2PKETF7EWN4BF4MSSL2Q2FM4CV4AA, seg_pos={0 0}, upload=450111]: Uploading piece... Dec 27 00:26:15.457423130 (512h57m15.328506165s) [upload=450111]: Next returned nil segment Dec 27 00:26:15.457443779 (512h57m15.328526815s) [upload=450111]: Waiting for error group managing segments... Dec 27 00:26:15.758087324 (512h57m15.629170359s) [noise=true , address=145.79.227.42:4122 , piece_id=I2KS77ZLJTKDKQY2ADJW3LGPN3PSXSNR3ZDQNEDISPXITN2ZWKPQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:15.767345973 (512h57m15.638429008s) [noise=true , address=145.79.227.37:4567 , piece_id=M7TFXHUWQDFNVHKIMCF5KFIGMPYCLKPBHSOKONSPWMA7F67RTLBQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:15.769246331 (512h57m15.640329366s) [noise=true , address=145.79.227.46:4601 , piece_id=IZ53F5AB6WVLSUAZZVCSKDW5GH6OHTPHVRSGM27OUJIHSAAK2BTA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:15.777540344 (512h57m15.648623379s) [noise=true , address=145.79.227.16:4911 , piece_id=WWR4HNZHSUAC5BUMODIRWKZJVS5XHGNY52HDXFGQ6I2WB76E2XIQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:15.843997747 (512h57m15.715080782s) [noise=true , address=145.79.227.41:4087 , piece_id=QXFO4PEXDZKP7LNVA7VBJQT3JPKWG4HUDEZHW5EY2NMVUV4FJORQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:15.961524649 (512h57m15.832607684s) [noise=true , address=45.91.116.8:4046 , piece_id=ZW7X5OSJRVOH6SRIW4SE6IHZSOTAHGPEV65OAIXRJHJO23SLK2QA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.010426414 (512h57m15.881509459s) [noise=true , address=45.91.116.5:4297 , piece_id=KP2KL5UU2H2ASJNCBWHC5DD2UME5TLFDCKCRBMVHNP4QAMIVM4UA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.016228541 (512h57m15.887311576s) [noise=true , address=77.90.31.11:4553 , piece_id=GVARHA2VBANU4DJZ2KPF46D2PKETF7EWN4BF4MSSL2Q2FM4CV4AA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.021859053 (512h57m15.892942098s) [noise=true , address=77.90.31.18:4805 , piece_id=GMBXRNG42MCB2VETHCZHTWQKE5JJMLQ4FKZILXPXKOJCPYABTNVQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.038785783 (512h57m15.909868818s) [noise=true , address=45.91.116.13:4155 , piece_id=2VRBWI6GMSR45ZZCTTY6CJAUQCXFBRYRPRUSCFIKHCEFPP3QHQ4A, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.135394756 (512h57m16.006477791s) [noise=true , address=45.91.116.10:4082 , piece_id=F3VILBFRIARIPRCI7FZ7T6UC3HF2OULRLY4B7LBNJKRIVV6ILYNQ, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.167091479 (512h57m16.038174525s) [noise=true , address=77.90.31.13:4627 , piece_id=BUEM2WTXD7F6H4ZJYCE5H5E7RXQHZBKYMEKT6FYJC2OAO6JBNZRA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.266401165 (512h57m16.13748421s) [noise=true , address=77.90.31.10:4517 , piece_id=RX2UH6EXJJKNQMMPHEG2ZOSR7NNYMTQ6OVR4S63RMCAKOYXJ5ZJA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: <nil> Dec 27 00:26:16.266419109 (512h57m16.137502144s) [seg_pos={0 0}, upload=450111]: Segment reached optimal threshold of 13 pieces. Dec 27 00:26:16.266618118 (512h57m16.137701414s) [noise=true , address=77.90.31.17:4773 , piece_id=IVXE3ZHEAF3JQO27NXEYP2EZHG3XXZRVXW3LXN5WNOH2SJY3KS2A, seg_pos={0 0}, upload=450111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:38764->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:38764->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38764->77.90.31.17:4773: use of closed network connection Dec 27 00:26:16.266629960 (512h57m16.137713006s) [noise=true , address=77.90.31.17:4773 , piece_id=IVXE3ZHEAF3JQO27NXEYP2EZHG3XXZRVXW3LXN5WNOH2SJY3KS2A, seg_pos={0 0}, upload=450111]: Long tail cancellation due to optimal threshold Dec 27 00:26:16.266907339 (512h57m16.137990384s) [noise=true , address=77.90.31.12:4590 , piece_id=CZCNJEG34EX7XIHMKE7YHABT7ITHDIYERGB6R2AL3CQVBZWZYV4Q, seg_pos={0 0}, upload=450111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 00:26:16.266914172 (512h57m16.137997207s) [noise=true , address=77.90.31.12:4590 , piece_id=CZCNJEG34EX7XIHMKE7YHABT7ITHDIYERGB6R2AL3CQVBZWZYV4Q, seg_pos={0 0}, upload=450111]: Long tail cancellation due to optimal threshold Dec 27 00:26:16.266988603 (512h57m16.138071648s) [noise=true , address=45.91.116.7:4009 , piece_id=NIIYPJLFDPQS6523FP2J3HMJRYFXPPGIQ76PSGDMCIMWVUZEDXZA, seg_pos={0 0}, upload=450111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 00:26:16.267005967 (512h57m16.138089002s) [noise=true , address=45.91.116.7:4009 , piece_id=NIIYPJLFDPQS6523FP2J3HMJRYFXPPGIQ76PSGDMCIMWVUZEDXZA, seg_pos={0 0}, upload=450111]: Long tail cancellation due to optimal threshold Dec 27 00:26:16.267054619 (512h57m16.138137654s) [seg_pos={0 0}, upload=450111]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:26:16.267163045 (512h57m16.13824608s) [upload=450111]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:26:16.267169428 (512h57m16.138252463s) [upload=450111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:26:16.556960039 (512h57m16.428043094s) [upload=450111]: Done uploading segments... Dec 27 00:26:16.556991860 (512h57m16.428074905s) [upload=450111]: Upload finished. err: <nil> Duration: 30.003130142s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 258.791013ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:26:18 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 258.828386ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXX..X.XX...X.........X.........X.......X.XX.XX.XXXX..X.XX] duration: 30.002046026s trace: https://jaeger.dev.storj.io/trace/41298950c8cc0dec FAIL (2025-12-27 00:26:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: Y57B5HMTXXTXCJ4IN2KVSTKYGEDZZBC6WYMV6LF5OAWI6W5YCTYQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: SXGDIWLK2NUWRS4WOZ5VM63UBHQY7FWF4OMCM7ASQULFDHY7HYOQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BSSANZWXXFFTWYUXEMTPWDJ5NIOKHU2Y7T3ICIUD4YUCN64AOREQ) context deadline exceeded ======LOGS====== ments... Dec 27 00:25:43.225981994 (512h56m43.097065039s) [upload=450102]: Got next segment. Inline: false Dec 27 00:25:43.226054853 (512h56m43.097137898s) [upload=450102]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:25:43.226060253 (512h56m43.097143288s) [upload=450102]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:25:43.471998344 (512h56m43.343081389s) [seg_pos={0 0}, upload=450102]: Begin upload segment... Dec 27 00:25:43.472060923 (512h56m43.343143968s) [seg_pos={0 0}, upload=450102]: Done begin upload segment. Dec 27 00:25:43.472083847 (512h56m43.343166882s) [noise=true , address=145.79.227.52:4447 , piece_id=MJP5IYUXZBCTAHDCS7EGJWDZJJS3QGU6H32PR433LGO4646OILSA, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.472097022 (512h56m43.343180057s) [noise=true , address=45.91.116.14:4189 , piece_id=BSSANZWXXFFTWYUXEMTPWDJ5NIOKHU2Y7T3ICIUD4YUCN64AOREQ, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.472592545 (512h56m43.34367559s) [noise=true , address=145.79.227.38:4977 , piece_id=G3JUY6IRPTB5EMUO7AMAIX766TXCECBJ3NUCRDGHZKLH47LENQCQ, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.473207186 (512h56m43.344290251s) [noise=true , address=145.79.227.13:4803 , piece_id=66GZSQAT2M3OR2SO4UQ76HUIO3NSQ5WCIOBWEFMYVYB34HJ4UUAQ, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.473620042 (512h56m43.344703077s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHMZ5GF6ARKTASNHMBGBLT6RUDSJQKYW5GHUA6TZWPBAYZDNEXA, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.474116146 (512h56m43.345199181s) [noise=true , address=145.79.227.11:4732 , piece_id=YL4JMROVHV36V2XEHNDXRQ6G4RFYV2UPJ2WLGT7WFKNWOBNGEN4Q, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.474647859 (512h56m43.345730894s) [noise=true , address=145.79.227.7:4295 , piece_id=WU6LVUQFULKZTRJ25F7U7D4VOWLQHGB5HUZDQQYNPR7BMKTNFS3A, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.475069612 (512h56m43.346152647s) [noise=true , address=145.79.227.45:4231 , piece_id=TCLEFETHLXIYFR5DSB6DIWCGOHRH6ZXZTWMLQ45SRI5BS4MM7SJQ, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.475400452 (512h56m43.346483497s) [noise=true , address=145.79.227.6:4257 , piece_id=WBRXCOZOOUDSWOOITZ6TSATXS5T4Z6RBB3YDFWLQCQDFOQSU6F6Q, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.475767470 (512h56m43.346850515s) [noise=true , address=145.79.227.49:4339 , piece_id=3FRNUE2JSQXXRQR3PLK5MEVRPOTGYWWG4FJ7GYHCIYOHQSJTV4NA, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.477176414 (512h56m43.348259449s) [noise=true , address=45.91.116.4:4939 , piece_id=YVNZ5PVFDUWMFF2YGNDUF26MY2ULGGFUKGJ5UF2SV5CZL3TQJHUA, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.478634641 (512h56m43.349717686s) [noise=true , address=77.90.31.11:4553 , piece_id=ZTNLXU67OWSAIPOL3HQLQZDAPBEM6TLWLWYWCLJYP6SHJSLE4PNQ, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.480151709 (512h56m43.351234744s) [noise=true , address=145.79.227.2:4114 , piece_id=EW3753XNNJL6T5IOLDE3KOZK44XF4PIZ5WYSSPEIGN3HMPPCSVBQ, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.481625365 (512h56m43.35270841s) [noise=true , address=45.91.116.13:4153 , piece_id=Y57B5HMTXXTXCJ4IN2KVSTKYGEDZZBC6WYMV6LF5OAWI6W5YCTYQ, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.482977189 (512h56m43.354060224s) [noise=true , address=145.79.227.42:4122 , piece_id=LR3JUOIDIZRSDRZGA3W5X7U3UDIKYTTM3DWZ2JLDWJVS6EGMHY6Q, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:43.484078896 (512h56m43.355161931s) [noise=true , address=77.90.31.12:4590 , piece_id=SXGDIWLK2NUWRS4WOZ5VM63UBHQY7FWF4OMCM7ASQULFDHY7HYOQ, seg_pos={0 0}, upload=450102]: Uploading piece... Dec 27 00:25:45.089670772 (512h56m44.960753817s) [upload=450102]: Next returned nil segment Dec 27 00:25:45.089685581 (512h56m44.960768626s) [upload=450102]: Waiting for error group managing segments... Dec 27 00:25:45.476589122 (512h56m45.347672157s) [noise=true , address=145.79.227.13:4803 , piece_id=66GZSQAT2M3OR2SO4UQ76HUIO3NSQ5WCIOBWEFMYVYB34HJ4UUAQ, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.476846042 (512h56m45.347929077s) [noise=true , address=145.79.227.52:4447 , piece_id=MJP5IYUXZBCTAHDCS7EGJWDZJJS3QGU6H32PR433LGO4646OILSA, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.486008889 (512h56m45.357091924s) [noise=true , address=145.79.227.11:4732 , piece_id=YL4JMROVHV36V2XEHNDXRQ6G4RFYV2UPJ2WLGT7WFKNWOBNGEN4Q, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.487151334 (512h56m45.358234379s) [noise=true , address=145.79.227.7:4295 , piece_id=WU6LVUQFULKZTRJ25F7U7D4VOWLQHGB5HUZDQQYNPR7BMKTNFS3A, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.498646753 (512h56m45.369729788s) [noise=true , address=145.79.227.49:4339 , piece_id=3FRNUE2JSQXXRQR3PLK5MEVRPOTGYWWG4FJ7GYHCIYOHQSJTV4NA, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.499912012 (512h56m45.370995057s) [noise=true , address=145.79.227.38:4977 , piece_id=G3JUY6IRPTB5EMUO7AMAIX766TXCECBJ3NUCRDGHZKLH47LENQCQ, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.505698670 (512h56m45.376781715s) [noise=true , address=145.79.227.42:4122 , piece_id=LR3JUOIDIZRSDRZGA3W5X7U3UDIKYTTM3DWZ2JLDWJVS6EGMHY6Q, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.510266168 (512h56m45.381349203s) [noise=true , address=145.79.227.45:4231 , piece_id=TCLEFETHLXIYFR5DSB6DIWCGOHRH6ZXZTWMLQ45SRI5BS4MM7SJQ, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.713730804 (512h56m45.584813839s) [noise=true , address=45.91.116.13:4153 , piece_id=Y57B5HMTXXTXCJ4IN2KVSTKYGEDZZBC6WYMV6LF5OAWI6W5YCTYQ, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.718749380 (512h56m45.589832415s) [noise=true , address=45.91.116.4:4939 , piece_id=YVNZ5PVFDUWMFF2YGNDUF26MY2ULGGFUKGJ5UF2SV5CZL3TQJHUA, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.728723582 (512h56m45.599806617s) [noise=true , address=45.91.116.14:4189 , piece_id=BSSANZWXXFFTWYUXEMTPWDJ5NIOKHU2Y7T3ICIUD4YUCN64AOREQ, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.745591690 (512h56m45.616674725s) [noise=true , address=77.90.31.12:4590 , piece_id=SXGDIWLK2NUWRS4WOZ5VM63UBHQY7FWF4OMCM7ASQULFDHY7HYOQ, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.808503862 (512h56m45.679586897s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHMZ5GF6ARKTASNHMBGBLT6RUDSJQKYW5GHUA6TZWPBAYZDNEXA, seg_pos={0 0}, upload=450102]: Done uploading piece. err: <nil> Dec 27 00:25:45.808523839 (512h56m45.679606874s) [seg_pos={0 0}, upload=450102]: Segment reached optimal threshold of 13 pieces. Dec 27 00:25:45.808691328 (512h56m45.679774363s) [noise=true , address=77.90.31.11:4553 , piece_id=ZTNLXU67OWSAIPOL3HQLQZDAPBEM6TLWLWYWCLJYP6SHJSLE4PNQ, seg_pos={0 0}, upload=450102]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:53286->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:53286->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53286->77.90.31.11:4553: use of closed network connection Dec 27 00:25:45.808696969 (512h56m45.679780014s) [noise=true , address=77.90.31.11:4553 , piece_id=ZTNLXU67OWSAIPOL3HQLQZDAPBEM6TLWLWYWCLJYP6SHJSLE4PNQ, seg_pos={0 0}, upload=450102]: Long tail cancellation due to optimal threshold Dec 27 00:25:45.808851874 (512h56m45.679934919s) [noise=true , address=145.79.227.6:4257 , piece_id=WBRXCOZOOUDSWOOITZ6TSATXS5T4Z6RBB3YDFWLQCQDFOQSU6F6Q, seg_pos={0 0}, upload=450102]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:48766->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:48766->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48766->145.79.227.6:4257: use of closed network connection Dec 27 00:25:45.808867714 (512h56m45.679950749s) [noise=true , address=145.79.227.6:4257 , piece_id=WBRXCOZOOUDSWOOITZ6TSATXS5T4Z6RBB3YDFWLQCQDFOQSU6F6Q, seg_pos={0 0}, upload=450102]: Long tail cancellation due to optimal threshold Dec 27 00:25:45.808981871 (512h56m45.680064906s) [noise=true , address=145.79.227.2:4114 , piece_id=EW3753XNNJL6T5IOLDE3KOZK44XF4PIZ5WYSSPEIGN3HMPPCSVBQ, seg_pos={0 0}, upload=450102]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: expected piece hash; context canceled; EOF Dec 27 00:25:45.808988645 (512h56m45.68007168s) [noise=true , address=145.79.227.2:4114 , piece_id=EW3753XNNJL6T5IOLDE3KOZK44XF4PIZ5WYSSPEIGN3HMPPCSVBQ, seg_pos={0 0}, upload=450102]: Long tail cancellation due to optimal threshold Dec 27 00:25:45.809006498 (512h56m45.680089533s) [seg_pos={0 0}, upload=450102]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:25:45.809069819 (512h56m45.680152864s) [upload=450102]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:25:45.809074558 (512h56m45.680157593s) [upload=450102]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:25:46.093654787 (512h56m45.964737842s) [upload=450102]: Done uploading segments... Dec 27 00:25:46.093677509 (512h56m45.964760544s) [upload=450102]: Upload finished. err: <nil> Duration: 30.002071293s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 260.156915ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:26:01 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 260.177755ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XXXX..X.XX...X.........X.........X.......X.XX.XX.XXXX..X.X] duration: 30.003785512s trace: https://jaeger.dev.storj.io/trace/221601c8c82f15b0 FAIL (2025-12-27 00:25:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 4KQW7EYYWT7PG7PSDERPPPM44IDLHLPVJTPMPXHVXRA5NST4WHSA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: VJIU4AABO2NSJKZEUYBEBC6WSMGHDPRBQMCEGHCMO5JTFDZZVTCQ) context deadline exceeded ======LOGS====== 27 00:24:56.642071697 (512h55m56.513154762s) [upload=450091]: Starting upload Dec 27 00:24:56.642213156 (512h55m56.513296211s) [upload=450091]: Done starting upload Dec 27 00:24:56.642318547 (512h55m56.513401612s) [upload=450091]: Uploading segments... Dec 27 00:24:56.642539096 (512h55m56.513622161s) [upload=450091]: Got next segment. Inline: false Dec 27 00:24:56.642585705 (512h55m56.51366874s) [upload=450091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:24:56.642591897 (512h55m56.513674962s) [upload=450091]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:24:56.887193523 (512h55m56.758276588s) [seg_pos={0 0}, upload=450091]: Begin upload segment... Dec 27 00:24:56.887248588 (512h55m56.758331633s) [seg_pos={0 0}, upload=450091]: Done begin upload segment. Dec 27 00:24:56.887283845 (512h55m56.75836688s) [noise=true , address=45.91.116.8:4045 , piece_id=ROXJHP5ZCQZ7ZC6J2TAVDITPFZGYK7ESLT4FFIKKKGHUYC64TQ3A, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.887353016 (512h55m56.758436072s) [noise=true , address=145.79.227.47:4267 , piece_id=47YG75HTIU2UJXWXQTKZM2ZZHVMIDYJRAPAAMZCAUDJUSX3ZIYCA, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.887917641 (512h55m56.759000676s) [noise=true , address=45.91.116.9:4264 , piece_id=YOFDIPBMADKANCU3AFO4XKUKQDMNVHULA6NKFFGAPM5ZKDD4C3JQ, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.888278589 (512h55m56.759361624s) [noise=true , address=145.79.227.52:4447 , piece_id=LD6C2YHG6NIBTOC7TUENFKI56OLSWHU4RKZ7VA2W4O5PFZ64JG4A, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.888286264 (512h55m56.759369299s) [noise=true , address=45.91.116.4:4937 , piece_id=5ZWPAFYPNWL6HLECP27O3SB4AA7O54TO4ZQ6P37JXE5EODWJ4SGA, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.889095965 (512h55m56.76017901s) [noise=true , address=45.91.116.5:4300 , piece_id=4KQW7EYYWT7PG7PSDERPPPM44IDLHLPVJTPMPXHVXRA5NST4WHSA, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.889482251 (512h55m56.760565286s) [noise=true , address=77.90.31.21:4877 , piece_id=K2UIHOVLZP64NTOYAU5PBINXN7HUTS7G5E46JBUUVZCGXS4OBSCA, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.889902842 (512h55m56.760985887s) [noise=true , address=145.79.227.39:4015 , piece_id=XGUNGRQARRDL3FOKAXDWQTBQUGZVS3U4QIKXPSJMV7OD66C3FV6A, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.890359782 (512h55m56.761442817s) [noise=true , address=145.79.227.16:4909 , piece_id=WFMEHU2XDHKMR34HLZWYZR3GISSRIYPKVKUY5VVWKX3QJEW2AFPQ, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.890664402 (512h55m56.761747437s) [noise=true , address=145.79.227.37:4565 , piece_id=4LJUG2473UQ6X5GA5LNANSS3FPBNEJX6NBJDALU5VHIB32V6XW7A, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.890970335 (512h55m56.76205337s) [noise=true , address=145.79.227.38:4978 , piece_id=RUIFBLD7ABW5IDAEE34REBSW2W4S6XHRKS4HMYOWC37CX5MV2MLA, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.891301495 (512h55m56.76238453s) [noise=true , address=45.91.116.12:4228 , piece_id=VJIU4AABO2NSJKZEUYBEBC6WSMGHDPRBQMCEGHCMO5JTFDZZVTCQ, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.891684815 (512h55m56.76276786s) [noise=true , address=145.79.227.15:4873 , piece_id=JUWAN45BKHTMSMQQ4NDAR2ZD2R5Z2XIRVXHF6FEKHFDCVRCUZI6Q, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.892929926 (512h55m56.764012961s) [noise=true , address=145.79.227.11:4731 , piece_id=YT6MXAK3KTP26L7YI4BPQZM2MEJ5MK76PFWZAWAZ7V7EFKTBW45A, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.901656352 (512h55m56.772739387s) [noise=true , address=145.79.227.49:4337 , piece_id=KVQX5VTR7E3FIHK4NTZ6NPQVS56YSWDS6IJP2W4KW6ESMQAZAFPQ, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:56.906325081 (512h55m56.777408126s) [noise=true , address=45.91.116.13:4154 , piece_id=5KW4CYFMSR25EYYCVSRRJGXBCAS4XE4ZUEW3KPLMQIDAS23XZLIQ, seg_pos={0 0}, upload=450091]: Uploading piece... Dec 27 00:24:58.614821556 (512h55m58.485904602s) [upload=450091]: Next returned nil segment Dec 27 00:24:58.614839952 (512h55m58.485922997s) [upload=450091]: Waiting for error group managing segments... Dec 27 00:24:58.906948134 (512h55m58.778031169s) [noise=true , address=145.79.227.47:4267 , piece_id=47YG75HTIU2UJXWXQTKZM2ZZHVMIDYJRAPAAMZCAUDJUSX3ZIYCA, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:58.909565227 (512h55m58.780648262s) [noise=true , address=145.79.227.49:4337 , piece_id=KVQX5VTR7E3FIHK4NTZ6NPQVS56YSWDS6IJP2W4KW6ESMQAZAFPQ, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:58.917891081 (512h55m58.788974116s) [noise=true , address=145.79.227.52:4447 , piece_id=LD6C2YHG6NIBTOC7TUENFKI56OLSWHU4RKZ7VA2W4O5PFZ64JG4A, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:58.918227631 (512h55m58.789310676s) [noise=true , address=145.79.227.11:4731 , piece_id=YT6MXAK3KTP26L7YI4BPQZM2MEJ5MK76PFWZAWAZ7V7EFKTBW45A, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:58.918844486 (512h55m58.789927521s) [noise=true , address=145.79.227.15:4873 , piece_id=JUWAN45BKHTMSMQQ4NDAR2ZD2R5Z2XIRVXHF6FEKHFDCVRCUZI6Q, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:58.919049476 (512h55m58.790132522s) [noise=true , address=145.79.227.38:4978 , piece_id=RUIFBLD7ABW5IDAEE34REBSW2W4S6XHRKS4HMYOWC37CX5MV2MLA, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:58.921319719 (512h55m58.792402754s) [noise=true , address=145.79.227.37:4565 , piece_id=4LJUG2473UQ6X5GA5LNANSS3FPBNEJX6NBJDALU5VHIB32V6XW7A, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:58.923951109 (512h55m58.795034154s) [noise=true , address=145.79.227.39:4015 , piece_id=XGUNGRQARRDL3FOKAXDWQTBQUGZVS3U4QIKXPSJMV7OD66C3FV6A, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:58.924578063 (512h55m58.795661108s) [noise=true , address=145.79.227.16:4909 , piece_id=WFMEHU2XDHKMR34HLZWYZR3GISSRIYPKVKUY5VVWKX3QJEW2AFPQ, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:59.141681048 (512h55m59.012764083s) [noise=true , address=45.91.116.12:4228 , piece_id=VJIU4AABO2NSJKZEUYBEBC6WSMGHDPRBQMCEGHCMO5JTFDZZVTCQ, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:59.145865988 (512h55m59.016949023s) [noise=true , address=45.91.116.9:4264 , piece_id=YOFDIPBMADKANCU3AFO4XKUKQDMNVHULA6NKFFGAPM5ZKDD4C3JQ, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:59.224985167 (512h55m59.096068202s) [noise=true , address=45.91.116.4:4937 , piece_id=5ZWPAFYPNWL6HLECP27O3SB4AA7O54TO4ZQ6P37JXE5EODWJ4SGA, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:59.572389601 (512h55m59.443472636s) [noise=true , address=45.91.116.5:4300 , piece_id=4KQW7EYYWT7PG7PSDERPPPM44IDLHLPVJTPMPXHVXRA5NST4WHSA, seg_pos={0 0}, upload=450091]: Done uploading piece. err: <nil> Dec 27 00:24:59.572410219 (512h55m59.443493254s) [seg_pos={0 0}, upload=450091]: Segment reached optimal threshold of 13 pieces. Dec 27 00:24:59.572615560 (512h55m59.443698615s) [noise=true , address=77.90.31.21:4877 , piece_id=K2UIHOVLZP64NTOYAU5PBINXN7HUTS7G5E46JBUUVZCGXS4OBSCA, seg_pos={0 0}, upload=450091]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 00:24:59.572624497 (512h55m59.443707532s) [noise=true , address=77.90.31.21:4877 , piece_id=K2UIHOVLZP64NTOYAU5PBINXN7HUTS7G5E46JBUUVZCGXS4OBSCA, seg_pos={0 0}, upload=450091]: Long tail cancellation due to optimal threshold Dec 27 00:24:59.572729477 (512h55m59.443812522s) [noise=true , address=45.91.116.8:4045 , piece_id=ROXJHP5ZCQZ7ZC6J2TAVDITPFZGYK7ESLT4FFIKKKGHUYC64TQ3A, seg_pos={0 0}, upload=450091]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 00:24:59.572745829 (512h55m59.443828874s) [noise=true , address=45.91.116.8:4045 , piece_id=ROXJHP5ZCQZ7ZC6J2TAVDITPFZGYK7ESLT4FFIKKKGHUYC64TQ3A, seg_pos={0 0}, upload=450091]: Long tail cancellation due to optimal threshold Dec 27 00:24:59.572751930 (512h55m59.443834965s) [noise=true , address=45.91.116.13:4154 , piece_id=5KW4CYFMSR25EYYCVSRRJGXBCAS4XE4ZUEW3KPLMQIDAS23XZLIQ, seg_pos={0 0}, upload=450091]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:52016->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:52016->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52016->45.91.116.13:4154: use of closed network connection Dec 27 00:24:59.572759655 (512h55m59.44384269s) [noise=true , address=45.91.116.13:4154 , piece_id=5KW4CYFMSR25EYYCVSRRJGXBCAS4XE4ZUEW3KPLMQIDAS23XZLIQ, seg_pos={0 0}, upload=450091]: Long tail cancellation due to optimal threshold Dec 27 00:24:59.572841491 (512h55m59.443924526s) [seg_pos={0 0}, upload=450091]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:24:59.572942474 (512h55m59.444025509s) [upload=450091]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:24:59.572948294 (512h55m59.444031329s) [upload=450091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:24:59.853234276 (512h55m59.724317311s) [upload=450091]: Done uploading segments... Dec 27 00:24:59.853255727 (512h55m59.724338762s) [upload=450091]: Upload finished. err: <nil> Duration: 30.003810368s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 258.023733ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:25:13 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 258.052998ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..XXXX..X.XX...X.........X.........X.......X.XX.XX.XXXX..X.] duration: 24.51644882s trace: https://jaeger.dev.storj.io/trace/2132a4fe773953b6 FAIL (2025-12-27 00:24:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: N4FSCRRIQSUASUEOI52OVIAISJLQQBJWAU6MQ6FAHVKSQH2HVF7A) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: PVY6L5HLCA55INLGEUWP55KP7MSINZJ3MIOE3YHX5OGHWSFMEKKQ) context deadline exceeded ======LOGS====== 27 00:23:26.060046224 (512h54m25.931129269s) [upload=450070]: Starting upload Dec 27 00:23:26.060098634 (512h54m25.931181669s) [upload=450070]: Done starting upload Dec 27 00:23:26.060273557 (512h54m25.931356602s) [upload=450070]: Uploading segments... Dec 27 00:23:26.060309235 (512h54m25.93139228s) [upload=450070]: Got next segment. Inline: false Dec 27 00:23:26.060336126 (512h54m25.931419161s) [upload=450070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:23:26.060341106 (512h54m25.931424141s) [upload=450070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:23:26.301681947 (512h54m26.172764983s) [seg_pos={0 0}, upload=450070]: Begin upload segment... Dec 27 00:23:26.301728546 (512h54m26.172811581s) [seg_pos={0 0}, upload=450070]: Done begin upload segment. Dec 27 00:23:26.301754486 (512h54m26.172837531s) [noise=true , address=145.79.227.45:4230 , piece_id=USRCT5C4BIRZQEPFSIOVQ3ERQOA3S4AQPPNVMIOVN7BHGSGIZW2A, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.302013408 (512h54m26.173096443s) [noise=true , address=77.90.31.18:4805 , piece_id=PVY6L5HLCA55INLGEUWP55KP7MSINZJ3MIOE3YHX5OGHWSFMEKKQ, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.302698633 (512h54m26.173781678s) [noise=true , address=45.91.116.13:4156 , piece_id=DJ4NB43L2ISDCYPMX4GVRH3TW3UHTWT2ITWLESBWNJGTUVMDVCOQ, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.303041907 (512h54m26.174124942s) [noise=true , address=145.79.227.52:4447 , piece_id=PSVDH6C52NFWB5IYRDIUADOCTKJNI6TIWNW3SC6FZEDYXHYNFYNQ, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.303373688 (512h54m26.174456733s) [noise=true , address=145.79.227.49:4339 , piece_id=Y3GDM7TBDDCQXVPLXVH3ULNQSYDKGFXSC22CCGTT2Z4SPJMI7O6A, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.303713395 (512h54m26.17479644s) [noise=true , address=145.79.227.50:4373 , piece_id=H6FEIRVG7WN55DB6VMHRRO2XGQFGEM4NBWOCLALUNBVAHXPTMNKA, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.304057250 (512h54m26.175140295s) [noise=true , address=45.91.116.8:4046 , piece_id=JBZBE5G5PAR2XNVP7K3W6CDPRWOCS7FYTZ7VEJHQ73YT4Z4RORDA, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.304398549 (512h54m26.175481594s) [noise=true , address=145.79.227.43:4157 , piece_id=TED6OAXPMK2CSL5MQE6TP5QVZHBQWF5GYJPKZJRDELWTMVVPCP6Q, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.304794193 (512h54m26.175877238s) [noise=true , address=145.79.227.36:4518 , piece_id=3GY3KWZP2PJJHXQN3IZLXASLFZ64O7PJVMILVH65LTOTBKLV4V2Q, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.305632580 (512h54m26.176715615s) [noise=true , address=77.90.31.12:4590 , piece_id=N4FSCRRIQSUASUEOI52OVIAISJLQQBJWAU6MQ6FAHVKSQH2HVF7A, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.306917026 (512h54m26.178000061s) [noise=true , address=145.79.227.6:4258 , piece_id=EMH6KE2KDWN7MOZOMPFNGDIXFDQQGJFQBEHZJR2Z6EY4EWIN3X4A, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.308910672 (512h54m26.179993717s) [noise=true , address=145.79.227.47:4265 , piece_id=TWZCAQC5CEWQMMXSTTKARAP77WGBY7ZXE566R3BKPC6GBU6LBVMA, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.310429403 (512h54m26.181512448s) [noise=true , address=145.79.227.41:4085 , piece_id=2PB7LWYLYGTFIE5TESM62VM72SCRRMO5MX3LUOXSDFPDRP67UXVQ, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.311671518 (512h54m26.182754563s) [noise=true , address=77.90.31.11:4556 , piece_id=RDAQRAWDATSEQHLLU37YALPFPCPDTCW2IP7YXML3OMQNSFLDXRZA, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.313935249 (512h54m26.185018284s) [noise=true , address=145.79.227.39:4015 , piece_id=J3JSFPH6GF6RFNIBS6XKL2JBORCOIYDG42KKBK3HDT5D2IMQR73A, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.317436064 (512h54m26.18851911s) [noise=true , address=145.79.227.42:4122 , piece_id=YIL2PBL6PBV2XG3QWQ4ORXT7EP6EP7NSP6WH3IRSY6WRKISLSOTQ, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:28.017782547 (512h54m27.888865593s) [upload=450070]: Next returned nil segment Dec 27 00:23:28.017842201 (512h54m27.888925246s) [upload=450070]: Waiting for error group managing segments... Dec 27 00:23:28.293578105 (512h54m28.16466116s) [noise=true , address=145.79.227.49:4339 , piece_id=Y3GDM7TBDDCQXVPLXVH3ULNQSYDKGFXSC22CCGTT2Z4SPJMI7O6A, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.306863963 (512h54m28.177946998s) [noise=true , address=145.79.227.50:4373 , piece_id=H6FEIRVG7WN55DB6VMHRRO2XGQFGEM4NBWOCLALUNBVAHXPTMNKA, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.309913910 (512h54m28.180996945s) [noise=true , address=145.79.227.36:4518 , piece_id=3GY3KWZP2PJJHXQN3IZLXASLFZ64O7PJVMILVH65LTOTBKLV4V2Q, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.321533897 (512h54m28.192616942s) [noise=true , address=145.79.227.45:4230 , piece_id=USRCT5C4BIRZQEPFSIOVQ3ERQOA3S4AQPPNVMIOVN7BHGSGIZW2A, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.326548875 (512h54m28.19763192s) [noise=true , address=145.79.227.39:4015 , piece_id=J3JSFPH6GF6RFNIBS6XKL2JBORCOIYDG42KKBK3HDT5D2IMQR73A, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.330429946 (512h54m28.201512981s) [noise=true , address=145.79.227.41:4085 , piece_id=2PB7LWYLYGTFIE5TESM62VM72SCRRMO5MX3LUOXSDFPDRP67UXVQ, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.331938959 (512h54m28.203022004s) [noise=true , address=145.79.227.52:4447 , piece_id=PSVDH6C52NFWB5IYRDIUADOCTKJNI6TIWNW3SC6FZEDYXHYNFYNQ, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.342016528 (512h54m28.213099573s) [noise=true , address=145.79.227.42:4122 , piece_id=YIL2PBL6PBV2XG3QWQ4ORXT7EP6EP7NSP6WH3IRSY6WRKISLSOTQ, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.356088432 (512h54m28.227171467s) [noise=true , address=145.79.227.47:4265 , piece_id=TWZCAQC5CEWQMMXSTTKARAP77WGBY7ZXE566R3BKPC6GBU6LBVMA, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.546866387 (512h54m28.417949422s) [noise=true , address=45.91.116.8:4046 , piece_id=JBZBE5G5PAR2XNVP7K3W6CDPRWOCS7FYTZ7VEJHQ73YT4Z4RORDA, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.570076571 (512h54m28.441159616s) [noise=true , address=77.90.31.18:4805 , piece_id=PVY6L5HLCA55INLGEUWP55KP7MSINZJ3MIOE3YHX5OGHWSFMEKKQ, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.583960758 (512h54m28.455043803s) [noise=true , address=77.90.31.12:4590 , piece_id=N4FSCRRIQSUASUEOI52OVIAISJLQQBJWAU6MQ6FAHVKSQH2HVF7A, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.606925757 (512h54m28.478008792s) [noise=true , address=45.91.116.13:4156 , piece_id=DJ4NB43L2ISDCYPMX4GVRH3TW3UHTWT2ITWLESBWNJGTUVMDVCOQ, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.606948280 (512h54m28.478031325s) [seg_pos={0 0}, upload=450070]: Segment reached optimal threshold of 13 pieces. Dec 27 00:23:28.607129896 (512h54m28.478212931s) [noise=true , address=145.79.227.43:4157 , piece_id=TED6OAXPMK2CSL5MQE6TP5QVZHBQWF5GYJPKZJRDELWTMVVPCP6Q, seg_pos={0 0}, upload=450070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 27 00:23:28.607135867 (512h54m28.478218902s) [noise=true , address=145.79.227.43:4157 , piece_id=TED6OAXPMK2CSL5MQE6TP5QVZHBQWF5GYJPKZJRDELWTMVVPCP6Q, seg_pos={0 0}, upload=450070]: Long tail cancellation due to optimal threshold Dec 27 00:23:28.607255735 (512h54m28.47833877s) [noise=true , address=145.79.227.6:4258 , piece_id=EMH6KE2KDWN7MOZOMPFNGDIXFDQQGJFQBEHZJR2Z6EY4EWIN3X4A, seg_pos={0 0}, upload=450070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:38892->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:38892->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38892->145.79.227.6:4258: use of closed network connection Dec 27 00:23:28.607261816 (512h54m28.478344861s) [noise=true , address=145.79.227.6:4258 , piece_id=EMH6KE2KDWN7MOZOMPFNGDIXFDQQGJFQBEHZJR2Z6EY4EWIN3X4A, seg_pos={0 0}, upload=450070]: Long tail cancellation due to optimal threshold Dec 27 00:23:28.607630669 (512h54m28.478713714s) [noise=true , address=77.90.31.11:4556 , piece_id=RDAQRAWDATSEQHLLU37YALPFPCPDTCW2IP7YXML3OMQNSFLDXRZA, seg_pos={0 0}, upload=450070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 00:23:28.607652340 (512h54m28.478735395s) [noise=true , address=77.90.31.11:4556 , piece_id=RDAQRAWDATSEQHLLU37YALPFPCPDTCW2IP7YXML3OMQNSFLDXRZA, seg_pos={0 0}, upload=450070]: Long tail cancellation due to optimal threshold Dec 27 00:23:28.607686806 (512h54m28.478769851s) [seg_pos={0 0}, upload=450070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:23:28.607809429 (512h54m28.478892474s) [upload=450070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:23:28.607817955 (512h54m28.47890099s) [upload=450070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:23:28.898406469 (512h54m28.769489514s) [upload=450070]: Done uploading segments... Dec 27 00:23:28.898429562 (512h54m28.769512597s) [upload=450070]: Upload finished. err: <nil> Duration: 24.516480199s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 266.108938ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:24:27 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 266.145638ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 257.336951ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:24:02 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 257.363241ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX..XXXX..X.XX...X.........X.........X.......X.XX.XX.XXXX..X] duration: 30.002721111s trace: https://jaeger.dev.storj.io/trace/2132a4fe773953b6 FAIL (2025-12-27 00:23:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: N4FSCRRIQSUASUEOI52OVIAISJLQQBJWAU6MQ6FAHVKSQH2HVF7A) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: PVY6L5HLCA55INLGEUWP55KP7MSINZJ3MIOE3YHX5OGHWSFMEKKQ) context deadline exceeded ======LOGS====== 27 00:23:26.060046224 (512h54m25.931129269s) [upload=450070]: Starting upload Dec 27 00:23:26.060098634 (512h54m25.931181669s) [upload=450070]: Done starting upload Dec 27 00:23:26.060273557 (512h54m25.931356602s) [upload=450070]: Uploading segments... Dec 27 00:23:26.060309235 (512h54m25.93139228s) [upload=450070]: Got next segment. Inline: false Dec 27 00:23:26.060336126 (512h54m25.931419161s) [upload=450070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:23:26.060341106 (512h54m25.931424141s) [upload=450070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:23:26.301681947 (512h54m26.172764983s) [seg_pos={0 0}, upload=450070]: Begin upload segment... Dec 27 00:23:26.301728546 (512h54m26.172811581s) [seg_pos={0 0}, upload=450070]: Done begin upload segment. Dec 27 00:23:26.301754486 (512h54m26.172837531s) [noise=true , address=145.79.227.45:4230 , piece_id=USRCT5C4BIRZQEPFSIOVQ3ERQOA3S4AQPPNVMIOVN7BHGSGIZW2A, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.302013408 (512h54m26.173096443s) [noise=true , address=77.90.31.18:4805 , piece_id=PVY6L5HLCA55INLGEUWP55KP7MSINZJ3MIOE3YHX5OGHWSFMEKKQ, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.302698633 (512h54m26.173781678s) [noise=true , address=45.91.116.13:4156 , piece_id=DJ4NB43L2ISDCYPMX4GVRH3TW3UHTWT2ITWLESBWNJGTUVMDVCOQ, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.303041907 (512h54m26.174124942s) [noise=true , address=145.79.227.52:4447 , piece_id=PSVDH6C52NFWB5IYRDIUADOCTKJNI6TIWNW3SC6FZEDYXHYNFYNQ, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.303373688 (512h54m26.174456733s) [noise=true , address=145.79.227.49:4339 , piece_id=Y3GDM7TBDDCQXVPLXVH3ULNQSYDKGFXSC22CCGTT2Z4SPJMI7O6A, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.303713395 (512h54m26.17479644s) [noise=true , address=145.79.227.50:4373 , piece_id=H6FEIRVG7WN55DB6VMHRRO2XGQFGEM4NBWOCLALUNBVAHXPTMNKA, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.304057250 (512h54m26.175140295s) [noise=true , address=45.91.116.8:4046 , piece_id=JBZBE5G5PAR2XNVP7K3W6CDPRWOCS7FYTZ7VEJHQ73YT4Z4RORDA, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.304398549 (512h54m26.175481594s) [noise=true , address=145.79.227.43:4157 , piece_id=TED6OAXPMK2CSL5MQE6TP5QVZHBQWF5GYJPKZJRDELWTMVVPCP6Q, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.304794193 (512h54m26.175877238s) [noise=true , address=145.79.227.36:4518 , piece_id=3GY3KWZP2PJJHXQN3IZLXASLFZ64O7PJVMILVH65LTOTBKLV4V2Q, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.305632580 (512h54m26.176715615s) [noise=true , address=77.90.31.12:4590 , piece_id=N4FSCRRIQSUASUEOI52OVIAISJLQQBJWAU6MQ6FAHVKSQH2HVF7A, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.306917026 (512h54m26.178000061s) [noise=true , address=145.79.227.6:4258 , piece_id=EMH6KE2KDWN7MOZOMPFNGDIXFDQQGJFQBEHZJR2Z6EY4EWIN3X4A, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.308910672 (512h54m26.179993717s) [noise=true , address=145.79.227.47:4265 , piece_id=TWZCAQC5CEWQMMXSTTKARAP77WGBY7ZXE566R3BKPC6GBU6LBVMA, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.310429403 (512h54m26.181512448s) [noise=true , address=145.79.227.41:4085 , piece_id=2PB7LWYLYGTFIE5TESM62VM72SCRRMO5MX3LUOXSDFPDRP67UXVQ, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.311671518 (512h54m26.182754563s) [noise=true , address=77.90.31.11:4556 , piece_id=RDAQRAWDATSEQHLLU37YALPFPCPDTCW2IP7YXML3OMQNSFLDXRZA, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.313935249 (512h54m26.185018284s) [noise=true , address=145.79.227.39:4015 , piece_id=J3JSFPH6GF6RFNIBS6XKL2JBORCOIYDG42KKBK3HDT5D2IMQR73A, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:26.317436064 (512h54m26.18851911s) [noise=true , address=145.79.227.42:4122 , piece_id=YIL2PBL6PBV2XG3QWQ4ORXT7EP6EP7NSP6WH3IRSY6WRKISLSOTQ, seg_pos={0 0}, upload=450070]: Uploading piece... Dec 27 00:23:28.017782547 (512h54m27.888865593s) [upload=450070]: Next returned nil segment Dec 27 00:23:28.017842201 (512h54m27.888925246s) [upload=450070]: Waiting for error group managing segments... Dec 27 00:23:28.293578105 (512h54m28.16466116s) [noise=true , address=145.79.227.49:4339 , piece_id=Y3GDM7TBDDCQXVPLXVH3ULNQSYDKGFXSC22CCGTT2Z4SPJMI7O6A, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.306863963 (512h54m28.177946998s) [noise=true , address=145.79.227.50:4373 , piece_id=H6FEIRVG7WN55DB6VMHRRO2XGQFGEM4NBWOCLALUNBVAHXPTMNKA, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.309913910 (512h54m28.180996945s) [noise=true , address=145.79.227.36:4518 , piece_id=3GY3KWZP2PJJHXQN3IZLXASLFZ64O7PJVMILVH65LTOTBKLV4V2Q, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.321533897 (512h54m28.192616942s) [noise=true , address=145.79.227.45:4230 , piece_id=USRCT5C4BIRZQEPFSIOVQ3ERQOA3S4AQPPNVMIOVN7BHGSGIZW2A, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.326548875 (512h54m28.19763192s) [noise=true , address=145.79.227.39:4015 , piece_id=J3JSFPH6GF6RFNIBS6XKL2JBORCOIYDG42KKBK3HDT5D2IMQR73A, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.330429946 (512h54m28.201512981s) [noise=true , address=145.79.227.41:4085 , piece_id=2PB7LWYLYGTFIE5TESM62VM72SCRRMO5MX3LUOXSDFPDRP67UXVQ, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.331938959 (512h54m28.203022004s) [noise=true , address=145.79.227.52:4447 , piece_id=PSVDH6C52NFWB5IYRDIUADOCTKJNI6TIWNW3SC6FZEDYXHYNFYNQ, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.342016528 (512h54m28.213099573s) [noise=true , address=145.79.227.42:4122 , piece_id=YIL2PBL6PBV2XG3QWQ4ORXT7EP6EP7NSP6WH3IRSY6WRKISLSOTQ, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.356088432 (512h54m28.227171467s) [noise=true , address=145.79.227.47:4265 , piece_id=TWZCAQC5CEWQMMXSTTKARAP77WGBY7ZXE566R3BKPC6GBU6LBVMA, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.546866387 (512h54m28.417949422s) [noise=true , address=45.91.116.8:4046 , piece_id=JBZBE5G5PAR2XNVP7K3W6CDPRWOCS7FYTZ7VEJHQ73YT4Z4RORDA, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.570076571 (512h54m28.441159616s) [noise=true , address=77.90.31.18:4805 , piece_id=PVY6L5HLCA55INLGEUWP55KP7MSINZJ3MIOE3YHX5OGHWSFMEKKQ, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.583960758 (512h54m28.455043803s) [noise=true , address=77.90.31.12:4590 , piece_id=N4FSCRRIQSUASUEOI52OVIAISJLQQBJWAU6MQ6FAHVKSQH2HVF7A, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.606925757 (512h54m28.478008792s) [noise=true , address=45.91.116.13:4156 , piece_id=DJ4NB43L2ISDCYPMX4GVRH3TW3UHTWT2ITWLESBWNJGTUVMDVCOQ, seg_pos={0 0}, upload=450070]: Done uploading piece. err: <nil> Dec 27 00:23:28.606948280 (512h54m28.478031325s) [seg_pos={0 0}, upload=450070]: Segment reached optimal threshold of 13 pieces. Dec 27 00:23:28.607129896 (512h54m28.478212931s) [noise=true , address=145.79.227.43:4157 , piece_id=TED6OAXPMK2CSL5MQE6TP5QVZHBQWF5GYJPKZJRDELWTMVVPCP6Q, seg_pos={0 0}, upload=450070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 27 00:23:28.607135867 (512h54m28.478218902s) [noise=true , address=145.79.227.43:4157 , piece_id=TED6OAXPMK2CSL5MQE6TP5QVZHBQWF5GYJPKZJRDELWTMVVPCP6Q, seg_pos={0 0}, upload=450070]: Long tail cancellation due to optimal threshold Dec 27 00:23:28.607255735 (512h54m28.47833877s) [noise=true , address=145.79.227.6:4258 , piece_id=EMH6KE2KDWN7MOZOMPFNGDIXFDQQGJFQBEHZJR2Z6EY4EWIN3X4A, seg_pos={0 0}, upload=450070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:38892->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:38892->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38892->145.79.227.6:4258: use of closed network connection Dec 27 00:23:28.607261816 (512h54m28.478344861s) [noise=true , address=145.79.227.6:4258 , piece_id=EMH6KE2KDWN7MOZOMPFNGDIXFDQQGJFQBEHZJR2Z6EY4EWIN3X4A, seg_pos={0 0}, upload=450070]: Long tail cancellation due to optimal threshold Dec 27 00:23:28.607630669 (512h54m28.478713714s) [noise=true , address=77.90.31.11:4556 , piece_id=RDAQRAWDATSEQHLLU37YALPFPCPDTCW2IP7YXML3OMQNSFLDXRZA, seg_pos={0 0}, upload=450070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 00:23:28.607652340 (512h54m28.478735395s) [noise=true , address=77.90.31.11:4556 , piece_id=RDAQRAWDATSEQHLLU37YALPFPCPDTCW2IP7YXML3OMQNSFLDXRZA, seg_pos={0 0}, upload=450070]: Long tail cancellation due to optimal threshold Dec 27 00:23:28.607686806 (512h54m28.478769851s) [seg_pos={0 0}, upload=450070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:23:28.607809429 (512h54m28.478892474s) [upload=450070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:23:28.607817955 (512h54m28.47890099s) [upload=450070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:23:28.898406469 (512h54m28.769489514s) [upload=450070]: Done uploading segments... Dec 27 00:23:28.898429562 (512h54m28.769512597s) [upload=450070]: Upload finished. err: <nil> Duration: 30.002745979s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 246.221575ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:23:33 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 246.243267ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XX..XXXX..X.XX...X.........X.........X.......X.XX.XX.XXXX..] duration: 15.657302895s trace: https://jaeger.dev.storj.io/trace/45e92f16d6f4f001 FAIL (2025-12-27 00:22:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 00:20:58.101294588 (512h51m57.972377643s) [upload=450039]: Starting upload Dec 27 00:20:58.101380792 (512h51m57.972463867s) [upload=450039]: Done starting upload Dec 27 00:20:58.101633353 (512h51m57.972716388s) [upload=450039]: Uploading segments... Dec 27 00:20:58.101737972 (512h51m57.972821017s) [upload=450039]: Got next segment. Inline: false Dec 27 00:20:58.101816241 (512h51m57.972899286s) [upload=450039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:20:58.101821611 (512h51m57.972904646s) [upload=450039]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:20:58.346048278 (512h51m58.217131313s) [seg_pos={0 0}, upload=450039]: Begin upload segment... Dec 27 00:20:58.346096229 (512h51m58.217179264s) [seg_pos={0 0}, upload=450039]: Done begin upload segment. Dec 27 00:20:58.346175060 (512h51m58.217258095s) [noise=true , address=45.91.116.6:4974 , piece_id=DSW4WPANNAQAENADCXQ5WQIPUFCWNLJTDVOQ3JJZ4YW2A4C6L2EA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.346856206 (512h51m58.217939251s) [noise=true , address=77.90.31.10:4520 , piece_id=C42JO6H63JGTK4H2HEAGCJQOXEFX5UO5FXA4SSASCWKMFFB2JCMA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.347133735 (512h51m58.21821677s) [noise=true , address=77.90.31.19:4842 , piece_id=K2JB5FVBFR375LHWEC5ELYEXGTOXXWNNIMLS4IDQWV4APQ3C35BA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.347406424 (512h51m58.218489459s) [noise=true , address=45.91.116.12:4227 , piece_id=U75CE757IHSPCC6YECAFNN6KSYBLC6NESQGZFU63ECX6IKYADAJQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.347805453 (512h51m58.218888488s) [noise=true , address=145.79.227.49:4338 , piece_id=KZRIIGT6NATMRGSHSYNBKY24SXI435SY7XYFQ4T2H5HNPIP3TFEQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.348094094 (512h51m58.219177129s) [noise=true , address=145.79.227.16:4911 , piece_id=CLMT4OLSXLYYWSRSJDJWZCVEO7GFNRA32I64EFWQYYULNH2FPE3Q, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.348378465 (512h51m58.2194615s) [noise=true , address=145.79.227.52:4447 , piece_id=7I3UPBC7K6PR36HAKDDU6U467SPBTQ753GVLOLEGA5CWQQVCWQCQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.348676422 (512h51m58.219759457s) [noise=true , address=77.90.31.16:4736 , piece_id=XSD75BL22WP5R2H37EX7HTBPAEIFBN7NK5FQWOMEJ4KJL4CTD6PA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.349054442 (512h51m58.220137487s) [noise=true , address=145.79.227.45:4231 , piece_id=YNDGAI7DWS6A6D6TXQOT3AW3ICYHXEUG76CDWTOGSLBRMCWOLW7Q, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.349436960 (512h51m58.220520005s) [noise=true , address=77.90.31.14:4663 , piece_id=WLCLUMO6AGTTP677Q26TV442BC4HP5L7P2LAMGE56UJAQ3PUD3XA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.350071098 (512h51m58.221154143s) [noise=true , address=45.91.116.11:4117 , piece_id=JKGNSLVGZLVNGRY22B5R7F43BW4S53NNSGDTV3JWPGFTK34PATWQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.350804975 (512h51m58.22188801s) [noise=true , address=45.91.116.8:4047 , piece_id=YVVFIVKQDPSLKMN7NHSF4KTRUVJWY53KXO6OHTZQ7RDALKBS763Q, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.351356125 (512h51m58.22243917s) [noise=true , address=145.79.227.47:4267 , piece_id=E4CTQBLA76PB2E3UTHZZUEA3PQJBBM7QMXMPEQ5JEMBEPRNR4QGA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.351900071 (512h51m58.222983116s) [noise=true , address=145.79.227.19:4079 , piece_id=LDNNYU4Q56BWQYQOSQNJMG5J2Q7F2RCKHCBLVYQKSQCJW5QUMIGA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.352403990 (512h51m58.223487035s) [noise=true , address=145.79.227.29:4942 , piece_id=3ZGGTKORYGELKNGSOCBAGA3QOWTEZNKDJELPMIJEWKB4UTIW5EBQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.354021199 (512h51m58.225104244s) [noise=true , address=45.91.116.14:4190 , piece_id=QT32BGKKH4DHLHXXYRWE3FCOQ37CSA47RYIFQPZH5OM7JQW22PZQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:21:00.071744428 (512h51m59.942827473s) [upload=450039]: Next returned nil segment Dec 27 00:21:00.071761681 (512h51m59.942844716s) [upload=450039]: Waiting for error group managing segments... Dec 27 00:21:00.348405734 (512h52m0.219488769s) [noise=true , address=145.79.227.52:4447 , piece_id=7I3UPBC7K6PR36HAKDDU6U467SPBTQ753GVLOLEGA5CWQQVCWQCQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.362661980 (512h52m0.233745015s) [noise=true , address=145.79.227.16:4911 , piece_id=CLMT4OLSXLYYWSRSJDJWZCVEO7GFNRA32I64EFWQYYULNH2FPE3Q, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.363828841 (512h52m0.234911876s) [noise=true , address=145.79.227.19:4079 , piece_id=LDNNYU4Q56BWQYQOSQNJMG5J2Q7F2RCKHCBLVYQKSQCJW5QUMIGA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.381387876 (512h52m0.252470911s) [noise=true , address=145.79.227.49:4338 , piece_id=KZRIIGT6NATMRGSHSYNBKY24SXI435SY7XYFQ4T2H5HNPIP3TFEQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.385071219 (512h52m0.256154264s) [noise=true , address=145.79.227.45:4231 , piece_id=YNDGAI7DWS6A6D6TXQOT3AW3ICYHXEUG76CDWTOGSLBRMCWOLW7Q, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.396581727 (512h52m0.267664772s) [noise=true , address=145.79.227.47:4267 , piece_id=E4CTQBLA76PB2E3UTHZZUEA3PQJBBM7QMXMPEQ5JEMBEPRNR4QGA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.529162386 (512h52m0.400245432s) [noise=true , address=77.90.31.19:4842 , piece_id=K2JB5FVBFR375LHWEC5ELYEXGTOXXWNNIMLS4IDQWV4APQ3C35BA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.539737112 (512h52m0.410820157s) [noise=true , address=145.79.227.29:4942 , piece_id=3ZGGTKORYGELKNGSOCBAGA3QOWTEZNKDJELPMIJEWKB4UTIW5EBQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.625108830 (512h52m0.496191875s) [noise=true , address=45.91.116.12:4227 , piece_id=U75CE757IHSPCC6YECAFNN6KSYBLC6NESQGZFU63ECX6IKYADAJQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.628177613 (512h52m0.499260648s) [noise=true , address=45.91.116.11:4117 , piece_id=JKGNSLVGZLVNGRY22B5R7F43BW4S53NNSGDTV3JWPGFTK34PATWQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.649261620 (512h52m0.520344665s) [noise=true , address=77.90.31.10:4520 , piece_id=C42JO6H63JGTK4H2HEAGCJQOXEFX5UO5FXA4SSASCWKMFFB2JCMA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.672944355 (512h52m0.5440274s) [noise=true , address=45.91.116.6:4974 , piece_id=DSW4WPANNAQAENADCXQ5WQIPUFCWNLJTDVOQ3JJZ4YW2A4C6L2EA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.701296711 (512h52m0.572379756s) [noise=true , address=45.91.116.14:4190 , piece_id=QT32BGKKH4DHLHXXYRWE3FCOQ37CSA47RYIFQPZH5OM7JQW22PZQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.701323171 (512h52m0.572406216s) [seg_pos={0 0}, upload=450039]: Segment reached optimal threshold of 13 pieces. Dec 27 00:21:00.701689348 (512h52m0.572772394s) [noise=true , address=77.90.31.16:4736 , piece_id=XSD75BL22WP5R2H37EX7HTBPAEIFBN7NK5FQWOMEJ4KJL4CTD6PA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:57956->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:57956->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57956->77.90.31.16:4736: use of closed network connection Dec 27 00:21:00.701698776 (512h52m0.572781821s) [noise=true , address=77.90.31.16:4736 , piece_id=XSD75BL22WP5R2H37EX7HTBPAEIFBN7NK5FQWOMEJ4KJL4CTD6PA, seg_pos={0 0}, upload=450039]: Long tail cancellation due to optimal threshold Dec 27 00:21:00.702100622 (512h52m0.573183667s) [noise=true , address=77.90.31.14:4663 , piece_id=WLCLUMO6AGTTP677Q26TV442BC4HP5L7P2LAMGE56UJAQ3PUD3XA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 27 00:21:00.702356819 (512h52m0.573439874s) [noise=true , address=77.90.31.14:4663 , piece_id=WLCLUMO6AGTTP677Q26TV442BC4HP5L7P2LAMGE56UJAQ3PUD3XA, seg_pos={0 0}, upload=450039]: Long tail cancellation due to optimal threshold Dec 27 00:21:00.702560808 (512h52m0.573643853s) [noise=true , address=45.91.116.8:4047 , piece_id=YVVFIVKQDPSLKMN7NHSF4KTRUVJWY53KXO6OHTZQ7RDALKBS763Q, seg_pos={0 0}, upload=450039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 00:21:00.702569975 (512h52m0.57365304s) [noise=true , address=45.91.116.8:4047 , piece_id=YVVFIVKQDPSLKMN7NHSF4KTRUVJWY53KXO6OHTZQ7RDALKBS763Q, seg_pos={0 0}, upload=450039]: Long tail cancellation due to optimal threshold Dec 27 00:21:00.702595855 (512h52m0.5736789s) [seg_pos={0 0}, upload=450039]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:21:00.702679213 (512h52m0.573762258s) [upload=450039]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:21:00.702686037 (512h52m0.573769082s) [upload=450039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:21:00.995394017 (512h52m0.866477062s) [upload=450039]: Done uploading segments... Dec 27 00:21:00.995413023 (512h52m0.866496068s) [upload=450039]: Upload finished. err: <nil> Duration: 15.657333042s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 245.650828ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:22:20 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 245.678491ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 244.09178ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:22:05 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 244.119643ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.XX..XXXX..X.XX...X.........X.........X.......X.XX.XX.XXXX.] duration: 6.112215584s trace: https://jaeger.dev.storj.io/trace/45e92f16d6f4f001 FAIL (2025-12-27 00:21:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 00:20:58.101294588 (512h51m57.972377643s) [upload=450039]: Starting upload Dec 27 00:20:58.101380792 (512h51m57.972463867s) [upload=450039]: Done starting upload Dec 27 00:20:58.101633353 (512h51m57.972716388s) [upload=450039]: Uploading segments... Dec 27 00:20:58.101737972 (512h51m57.972821017s) [upload=450039]: Got next segment. Inline: false Dec 27 00:20:58.101816241 (512h51m57.972899286s) [upload=450039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:20:58.101821611 (512h51m57.972904646s) [upload=450039]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:20:58.346048278 (512h51m58.217131313s) [seg_pos={0 0}, upload=450039]: Begin upload segment... Dec 27 00:20:58.346096229 (512h51m58.217179264s) [seg_pos={0 0}, upload=450039]: Done begin upload segment. Dec 27 00:20:58.346175060 (512h51m58.217258095s) [noise=true , address=45.91.116.6:4974 , piece_id=DSW4WPANNAQAENADCXQ5WQIPUFCWNLJTDVOQ3JJZ4YW2A4C6L2EA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.346856206 (512h51m58.217939251s) [noise=true , address=77.90.31.10:4520 , piece_id=C42JO6H63JGTK4H2HEAGCJQOXEFX5UO5FXA4SSASCWKMFFB2JCMA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.347133735 (512h51m58.21821677s) [noise=true , address=77.90.31.19:4842 , piece_id=K2JB5FVBFR375LHWEC5ELYEXGTOXXWNNIMLS4IDQWV4APQ3C35BA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.347406424 (512h51m58.218489459s) [noise=true , address=45.91.116.12:4227 , piece_id=U75CE757IHSPCC6YECAFNN6KSYBLC6NESQGZFU63ECX6IKYADAJQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.347805453 (512h51m58.218888488s) [noise=true , address=145.79.227.49:4338 , piece_id=KZRIIGT6NATMRGSHSYNBKY24SXI435SY7XYFQ4T2H5HNPIP3TFEQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.348094094 (512h51m58.219177129s) [noise=true , address=145.79.227.16:4911 , piece_id=CLMT4OLSXLYYWSRSJDJWZCVEO7GFNRA32I64EFWQYYULNH2FPE3Q, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.348378465 (512h51m58.2194615s) [noise=true , address=145.79.227.52:4447 , piece_id=7I3UPBC7K6PR36HAKDDU6U467SPBTQ753GVLOLEGA5CWQQVCWQCQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.348676422 (512h51m58.219759457s) [noise=true , address=77.90.31.16:4736 , piece_id=XSD75BL22WP5R2H37EX7HTBPAEIFBN7NK5FQWOMEJ4KJL4CTD6PA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.349054442 (512h51m58.220137487s) [noise=true , address=145.79.227.45:4231 , piece_id=YNDGAI7DWS6A6D6TXQOT3AW3ICYHXEUG76CDWTOGSLBRMCWOLW7Q, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.349436960 (512h51m58.220520005s) [noise=true , address=77.90.31.14:4663 , piece_id=WLCLUMO6AGTTP677Q26TV442BC4HP5L7P2LAMGE56UJAQ3PUD3XA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.350071098 (512h51m58.221154143s) [noise=true , address=45.91.116.11:4117 , piece_id=JKGNSLVGZLVNGRY22B5R7F43BW4S53NNSGDTV3JWPGFTK34PATWQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.350804975 (512h51m58.22188801s) [noise=true , address=45.91.116.8:4047 , piece_id=YVVFIVKQDPSLKMN7NHSF4KTRUVJWY53KXO6OHTZQ7RDALKBS763Q, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.351356125 (512h51m58.22243917s) [noise=true , address=145.79.227.47:4267 , piece_id=E4CTQBLA76PB2E3UTHZZUEA3PQJBBM7QMXMPEQ5JEMBEPRNR4QGA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.351900071 (512h51m58.222983116s) [noise=true , address=145.79.227.19:4079 , piece_id=LDNNYU4Q56BWQYQOSQNJMG5J2Q7F2RCKHCBLVYQKSQCJW5QUMIGA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.352403990 (512h51m58.223487035s) [noise=true , address=145.79.227.29:4942 , piece_id=3ZGGTKORYGELKNGSOCBAGA3QOWTEZNKDJELPMIJEWKB4UTIW5EBQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.354021199 (512h51m58.225104244s) [noise=true , address=45.91.116.14:4190 , piece_id=QT32BGKKH4DHLHXXYRWE3FCOQ37CSA47RYIFQPZH5OM7JQW22PZQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:21:00.071744428 (512h51m59.942827473s) [upload=450039]: Next returned nil segment Dec 27 00:21:00.071761681 (512h51m59.942844716s) [upload=450039]: Waiting for error group managing segments... Dec 27 00:21:00.348405734 (512h52m0.219488769s) [noise=true , address=145.79.227.52:4447 , piece_id=7I3UPBC7K6PR36HAKDDU6U467SPBTQ753GVLOLEGA5CWQQVCWQCQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.362661980 (512h52m0.233745015s) [noise=true , address=145.79.227.16:4911 , piece_id=CLMT4OLSXLYYWSRSJDJWZCVEO7GFNRA32I64EFWQYYULNH2FPE3Q, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.363828841 (512h52m0.234911876s) [noise=true , address=145.79.227.19:4079 , piece_id=LDNNYU4Q56BWQYQOSQNJMG5J2Q7F2RCKHCBLVYQKSQCJW5QUMIGA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.381387876 (512h52m0.252470911s) [noise=true , address=145.79.227.49:4338 , piece_id=KZRIIGT6NATMRGSHSYNBKY24SXI435SY7XYFQ4T2H5HNPIP3TFEQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.385071219 (512h52m0.256154264s) [noise=true , address=145.79.227.45:4231 , piece_id=YNDGAI7DWS6A6D6TXQOT3AW3ICYHXEUG76CDWTOGSLBRMCWOLW7Q, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.396581727 (512h52m0.267664772s) [noise=true , address=145.79.227.47:4267 , piece_id=E4CTQBLA76PB2E3UTHZZUEA3PQJBBM7QMXMPEQ5JEMBEPRNR4QGA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.529162386 (512h52m0.400245432s) [noise=true , address=77.90.31.19:4842 , piece_id=K2JB5FVBFR375LHWEC5ELYEXGTOXXWNNIMLS4IDQWV4APQ3C35BA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.539737112 (512h52m0.410820157s) [noise=true , address=145.79.227.29:4942 , piece_id=3ZGGTKORYGELKNGSOCBAGA3QOWTEZNKDJELPMIJEWKB4UTIW5EBQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.625108830 (512h52m0.496191875s) [noise=true , address=45.91.116.12:4227 , piece_id=U75CE757IHSPCC6YECAFNN6KSYBLC6NESQGZFU63ECX6IKYADAJQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.628177613 (512h52m0.499260648s) [noise=true , address=45.91.116.11:4117 , piece_id=JKGNSLVGZLVNGRY22B5R7F43BW4S53NNSGDTV3JWPGFTK34PATWQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.649261620 (512h52m0.520344665s) [noise=true , address=77.90.31.10:4520 , piece_id=C42JO6H63JGTK4H2HEAGCJQOXEFX5UO5FXA4SSASCWKMFFB2JCMA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.672944355 (512h52m0.5440274s) [noise=true , address=45.91.116.6:4974 , piece_id=DSW4WPANNAQAENADCXQ5WQIPUFCWNLJTDVOQ3JJZ4YW2A4C6L2EA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.701296711 (512h52m0.572379756s) [noise=true , address=45.91.116.14:4190 , piece_id=QT32BGKKH4DHLHXXYRWE3FCOQ37CSA47RYIFQPZH5OM7JQW22PZQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.701323171 (512h52m0.572406216s) [seg_pos={0 0}, upload=450039]: Segment reached optimal threshold of 13 pieces. Dec 27 00:21:00.701689348 (512h52m0.572772394s) [noise=true , address=77.90.31.16:4736 , piece_id=XSD75BL22WP5R2H37EX7HTBPAEIFBN7NK5FQWOMEJ4KJL4CTD6PA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:57956->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:57956->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57956->77.90.31.16:4736: use of closed network connection Dec 27 00:21:00.701698776 (512h52m0.572781821s) [noise=true , address=77.90.31.16:4736 , piece_id=XSD75BL22WP5R2H37EX7HTBPAEIFBN7NK5FQWOMEJ4KJL4CTD6PA, seg_pos={0 0}, upload=450039]: Long tail cancellation due to optimal threshold Dec 27 00:21:00.702100622 (512h52m0.573183667s) [noise=true , address=77.90.31.14:4663 , piece_id=WLCLUMO6AGTTP677Q26TV442BC4HP5L7P2LAMGE56UJAQ3PUD3XA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 27 00:21:00.702356819 (512h52m0.573439874s) [noise=true , address=77.90.31.14:4663 , piece_id=WLCLUMO6AGTTP677Q26TV442BC4HP5L7P2LAMGE56UJAQ3PUD3XA, seg_pos={0 0}, upload=450039]: Long tail cancellation due to optimal threshold Dec 27 00:21:00.702560808 (512h52m0.573643853s) [noise=true , address=45.91.116.8:4047 , piece_id=YVVFIVKQDPSLKMN7NHSF4KTRUVJWY53KXO6OHTZQ7RDALKBS763Q, seg_pos={0 0}, upload=450039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 00:21:00.702569975 (512h52m0.57365304s) [noise=true , address=45.91.116.8:4047 , piece_id=YVVFIVKQDPSLKMN7NHSF4KTRUVJWY53KXO6OHTZQ7RDALKBS763Q, seg_pos={0 0}, upload=450039]: Long tail cancellation due to optimal threshold Dec 27 00:21:00.702595855 (512h52m0.5736789s) [seg_pos={0 0}, upload=450039]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:21:00.702679213 (512h52m0.573762258s) [upload=450039]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:21:00.702686037 (512h52m0.573769082s) [upload=450039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:21:00.995394017 (512h52m0.866477062s) [upload=450039]: Done uploading segments... Dec 27 00:21:00.995413023 (512h52m0.866496068s) [upload=450039]: Upload finished. err: <nil> Duration: 6.112246312s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 249.652007ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:21:40 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 249.681373ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX.XX..XXXX..X.XX...X.........X.........X.......X.XX.XX.XXXX] duration: 30.002618587s trace: https://jaeger.dev.storj.io/trace/45e92f16d6f4f001 FAIL (2025-12-27 00:21:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 00:20:58.101294588 (512h51m57.972377643s) [upload=450039]: Starting upload Dec 27 00:20:58.101380792 (512h51m57.972463867s) [upload=450039]: Done starting upload Dec 27 00:20:58.101633353 (512h51m57.972716388s) [upload=450039]: Uploading segments... Dec 27 00:20:58.101737972 (512h51m57.972821017s) [upload=450039]: Got next segment. Inline: false Dec 27 00:20:58.101816241 (512h51m57.972899286s) [upload=450039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:20:58.101821611 (512h51m57.972904646s) [upload=450039]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:20:58.346048278 (512h51m58.217131313s) [seg_pos={0 0}, upload=450039]: Begin upload segment... Dec 27 00:20:58.346096229 (512h51m58.217179264s) [seg_pos={0 0}, upload=450039]: Done begin upload segment. Dec 27 00:20:58.346175060 (512h51m58.217258095s) [noise=true , address=45.91.116.6:4974 , piece_id=DSW4WPANNAQAENADCXQ5WQIPUFCWNLJTDVOQ3JJZ4YW2A4C6L2EA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.346856206 (512h51m58.217939251s) [noise=true , address=77.90.31.10:4520 , piece_id=C42JO6H63JGTK4H2HEAGCJQOXEFX5UO5FXA4SSASCWKMFFB2JCMA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.347133735 (512h51m58.21821677s) [noise=true , address=77.90.31.19:4842 , piece_id=K2JB5FVBFR375LHWEC5ELYEXGTOXXWNNIMLS4IDQWV4APQ3C35BA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.347406424 (512h51m58.218489459s) [noise=true , address=45.91.116.12:4227 , piece_id=U75CE757IHSPCC6YECAFNN6KSYBLC6NESQGZFU63ECX6IKYADAJQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.347805453 (512h51m58.218888488s) [noise=true , address=145.79.227.49:4338 , piece_id=KZRIIGT6NATMRGSHSYNBKY24SXI435SY7XYFQ4T2H5HNPIP3TFEQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.348094094 (512h51m58.219177129s) [noise=true , address=145.79.227.16:4911 , piece_id=CLMT4OLSXLYYWSRSJDJWZCVEO7GFNRA32I64EFWQYYULNH2FPE3Q, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.348378465 (512h51m58.2194615s) [noise=true , address=145.79.227.52:4447 , piece_id=7I3UPBC7K6PR36HAKDDU6U467SPBTQ753GVLOLEGA5CWQQVCWQCQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.348676422 (512h51m58.219759457s) [noise=true , address=77.90.31.16:4736 , piece_id=XSD75BL22WP5R2H37EX7HTBPAEIFBN7NK5FQWOMEJ4KJL4CTD6PA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.349054442 (512h51m58.220137487s) [noise=true , address=145.79.227.45:4231 , piece_id=YNDGAI7DWS6A6D6TXQOT3AW3ICYHXEUG76CDWTOGSLBRMCWOLW7Q, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.349436960 (512h51m58.220520005s) [noise=true , address=77.90.31.14:4663 , piece_id=WLCLUMO6AGTTP677Q26TV442BC4HP5L7P2LAMGE56UJAQ3PUD3XA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.350071098 (512h51m58.221154143s) [noise=true , address=45.91.116.11:4117 , piece_id=JKGNSLVGZLVNGRY22B5R7F43BW4S53NNSGDTV3JWPGFTK34PATWQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.350804975 (512h51m58.22188801s) [noise=true , address=45.91.116.8:4047 , piece_id=YVVFIVKQDPSLKMN7NHSF4KTRUVJWY53KXO6OHTZQ7RDALKBS763Q, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.351356125 (512h51m58.22243917s) [noise=true , address=145.79.227.47:4267 , piece_id=E4CTQBLA76PB2E3UTHZZUEA3PQJBBM7QMXMPEQ5JEMBEPRNR4QGA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.351900071 (512h51m58.222983116s) [noise=true , address=145.79.227.19:4079 , piece_id=LDNNYU4Q56BWQYQOSQNJMG5J2Q7F2RCKHCBLVYQKSQCJW5QUMIGA, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.352403990 (512h51m58.223487035s) [noise=true , address=145.79.227.29:4942 , piece_id=3ZGGTKORYGELKNGSOCBAGA3QOWTEZNKDJELPMIJEWKB4UTIW5EBQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:20:58.354021199 (512h51m58.225104244s) [noise=true , address=45.91.116.14:4190 , piece_id=QT32BGKKH4DHLHXXYRWE3FCOQ37CSA47RYIFQPZH5OM7JQW22PZQ, seg_pos={0 0}, upload=450039]: Uploading piece... Dec 27 00:21:00.071744428 (512h51m59.942827473s) [upload=450039]: Next returned nil segment Dec 27 00:21:00.071761681 (512h51m59.942844716s) [upload=450039]: Waiting for error group managing segments... Dec 27 00:21:00.348405734 (512h52m0.219488769s) [noise=true , address=145.79.227.52:4447 , piece_id=7I3UPBC7K6PR36HAKDDU6U467SPBTQ753GVLOLEGA5CWQQVCWQCQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.362661980 (512h52m0.233745015s) [noise=true , address=145.79.227.16:4911 , piece_id=CLMT4OLSXLYYWSRSJDJWZCVEO7GFNRA32I64EFWQYYULNH2FPE3Q, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.363828841 (512h52m0.234911876s) [noise=true , address=145.79.227.19:4079 , piece_id=LDNNYU4Q56BWQYQOSQNJMG5J2Q7F2RCKHCBLVYQKSQCJW5QUMIGA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.381387876 (512h52m0.252470911s) [noise=true , address=145.79.227.49:4338 , piece_id=KZRIIGT6NATMRGSHSYNBKY24SXI435SY7XYFQ4T2H5HNPIP3TFEQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.385071219 (512h52m0.256154264s) [noise=true , address=145.79.227.45:4231 , piece_id=YNDGAI7DWS6A6D6TXQOT3AW3ICYHXEUG76CDWTOGSLBRMCWOLW7Q, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.396581727 (512h52m0.267664772s) [noise=true , address=145.79.227.47:4267 , piece_id=E4CTQBLA76PB2E3UTHZZUEA3PQJBBM7QMXMPEQ5JEMBEPRNR4QGA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.529162386 (512h52m0.400245432s) [noise=true , address=77.90.31.19:4842 , piece_id=K2JB5FVBFR375LHWEC5ELYEXGTOXXWNNIMLS4IDQWV4APQ3C35BA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.539737112 (512h52m0.410820157s) [noise=true , address=145.79.227.29:4942 , piece_id=3ZGGTKORYGELKNGSOCBAGA3QOWTEZNKDJELPMIJEWKB4UTIW5EBQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.625108830 (512h52m0.496191875s) [noise=true , address=45.91.116.12:4227 , piece_id=U75CE757IHSPCC6YECAFNN6KSYBLC6NESQGZFU63ECX6IKYADAJQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.628177613 (512h52m0.499260648s) [noise=true , address=45.91.116.11:4117 , piece_id=JKGNSLVGZLVNGRY22B5R7F43BW4S53NNSGDTV3JWPGFTK34PATWQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.649261620 (512h52m0.520344665s) [noise=true , address=77.90.31.10:4520 , piece_id=C42JO6H63JGTK4H2HEAGCJQOXEFX5UO5FXA4SSASCWKMFFB2JCMA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.672944355 (512h52m0.5440274s) [noise=true , address=45.91.116.6:4974 , piece_id=DSW4WPANNAQAENADCXQ5WQIPUFCWNLJTDVOQ3JJZ4YW2A4C6L2EA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.701296711 (512h52m0.572379756s) [noise=true , address=45.91.116.14:4190 , piece_id=QT32BGKKH4DHLHXXYRWE3FCOQ37CSA47RYIFQPZH5OM7JQW22PZQ, seg_pos={0 0}, upload=450039]: Done uploading piece. err: <nil> Dec 27 00:21:00.701323171 (512h52m0.572406216s) [seg_pos={0 0}, upload=450039]: Segment reached optimal threshold of 13 pieces. Dec 27 00:21:00.701689348 (512h52m0.572772394s) [noise=true , address=77.90.31.16:4736 , piece_id=XSD75BL22WP5R2H37EX7HTBPAEIFBN7NK5FQWOMEJ4KJL4CTD6PA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:57956->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:57956->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57956->77.90.31.16:4736: use of closed network connection Dec 27 00:21:00.701698776 (512h52m0.572781821s) [noise=true , address=77.90.31.16:4736 , piece_id=XSD75BL22WP5R2H37EX7HTBPAEIFBN7NK5FQWOMEJ4KJL4CTD6PA, seg_pos={0 0}, upload=450039]: Long tail cancellation due to optimal threshold Dec 27 00:21:00.702100622 (512h52m0.573183667s) [noise=true , address=77.90.31.14:4663 , piece_id=WLCLUMO6AGTTP677Q26TV442BC4HP5L7P2LAMGE56UJAQ3PUD3XA, seg_pos={0 0}, upload=450039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 27 00:21:00.702356819 (512h52m0.573439874s) [noise=true , address=77.90.31.14:4663 , piece_id=WLCLUMO6AGTTP677Q26TV442BC4HP5L7P2LAMGE56UJAQ3PUD3XA, seg_pos={0 0}, upload=450039]: Long tail cancellation due to optimal threshold Dec 27 00:21:00.702560808 (512h52m0.573643853s) [noise=true , address=45.91.116.8:4047 , piece_id=YVVFIVKQDPSLKMN7NHSF4KTRUVJWY53KXO6OHTZQ7RDALKBS763Q, seg_pos={0 0}, upload=450039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 00:21:00.702569975 (512h52m0.57365304s) [noise=true , address=45.91.116.8:4047 , piece_id=YVVFIVKQDPSLKMN7NHSF4KTRUVJWY53KXO6OHTZQ7RDALKBS763Q, seg_pos={0 0}, upload=450039]: Long tail cancellation due to optimal threshold Dec 27 00:21:00.702595855 (512h52m0.5736789s) [seg_pos={0 0}, upload=450039]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:21:00.702679213 (512h52m0.573762258s) [upload=450039]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:21:00.702686037 (512h52m0.573769082s) [upload=450039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:21:00.995394017 (512h52m0.866477062s) [upload=450039]: Done uploading segments... Dec 27 00:21:00.995413023 (512h52m0.866496068s) [upload=450039]: Upload finished. err: <nil> Duration: 30.002638975s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 257.061415ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:20:52 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 257.112502ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 280.714293ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:20:28 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 280.737267ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX.XX..XXXX..X.XX...X.........X.........X.......X.XX.XX.XXX] duration: 30.002995415s trace: https://jaeger.dev.storj.io/trace/2c409fe778f41ba2 FAIL (2025-12-27 00:20:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 00:19:49.179991907 (512h50m49.051074952s) [upload=450021]: Starting upload Dec 27 00:19:49.180075167 (512h50m49.051158212s) [upload=450021]: Done starting upload Dec 27 00:19:49.180131002 (512h50m49.051214047s) [upload=450021]: Uploading segments... Dec 27 00:19:49.180400856 (512h50m49.051483901s) [upload=450021]: Got next segment. Inline: false Dec 27 00:19:49.180442616 (512h50m49.051525661s) [upload=450021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:19:49.180448778 (512h50m49.051531823s) [upload=450021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:19:49.423001336 (512h50m49.294084372s) [seg_pos={0 0}, upload=450021]: Begin upload segment... Dec 27 00:19:49.423053336 (512h50m49.294136371s) [seg_pos={0 0}, upload=450021]: Done begin upload segment. Dec 27 00:19:49.423080298 (512h50m49.294163333s) [noise=true , address=145.79.227.36:4518 , piece_id=F6TIEVNRV7265TAKZHBV3AA3TUYQKU5G4W2MUCOOK5275XTSXCVQ, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.423190197 (512h50m49.294273242s) [noise=true , address=45.91.116.14:4189 , piece_id=X3TU6TUQ5AOKLJMBA3BX7EBJ5YANCIRWFXN7MOAU26MV3XHD242Q, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.423625916 (512h50m49.294708951s) [noise=true , address=145.79.227.46:4602 , piece_id=R7LUXLSCEY4CZWSX3Y2DE7BTMMKFISLFLVJERCPF7VD5OEIXAENQ, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.424224536 (512h50m49.295307581s) [noise=true , address=145.79.227.42:4122 , piece_id=PCDQN2P5EMM2PRUKHXBNMJFWCRQLQPBILUSQPE4JUXRHCK7UX3YA, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.424884091 (512h50m49.295967147s) [noise=true , address=145.79.227.2:4114 , piece_id=HRBXCRTY4F2KQYPWDRYDCRV3JIS25FGKPUHE23QRUKH7OY74MWQA, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.425149938 (512h50m49.296232983s) [noise=true , address=145.79.227.38:4978 , piece_id=UE5ZHWVK2CCNX7MBF7PLQ4VRGTXWW2VZYWUE2T32XMGGZ3HDB4XA, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.425772043 (512h50m49.296855088s) [noise=true , address=145.79.227.12:4765 , piece_id=KRGHSNDXMBJAXQJDJPRGC2CECBY472ONWHEG7WTMW3YD55A3AFFQ, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.426062395 (512h50m49.29714544s) [noise=true , address=145.79.227.13:4804 , piece_id=JF2I4CDZ737IDWIDA2D43NQLHBE6C634HMCVOBRMS4MKKS4LB73Q, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.426982788 (512h50m49.298065824s) [noise=true , address=77.90.31.17:4773 , piece_id=GNJNZSSRMWGGFJ3LVAONWNTULVO6X4OQW3NKHVSC7PC76RPEZ47Q, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.427073711 (512h50m49.298156756s) [noise=true , address=45.91.116.8:4045 , piece_id=KJ3D3HTVI4L5GXWYXEKKDTBCXADH6JCGVOIYPLFBCHT3H5FZRSKA, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.428104314 (512h50m49.299187349s) [noise=true , address=77.90.31.10:4519 , piece_id=X6ALESIWGNAFPYETMJMQY6NGBFTBSBOSDSCJNGX75POL2GL6SBSA, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.428178074 (512h50m49.299261119s) [noise=true , address=145.79.227.50:4373 , piece_id=M42YYUSUI2XL7LBMI5RT65VOWYIYFC7XFE6JFKPMWP33ESSMXXMQ, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.429753574 (512h50m49.300836609s) [noise=true , address=45.91.116.11:4118 , piece_id=MRENDTVHTQU77N2OKS2LL6TRU2IZGPASCCIJI2BUYBS4LD33CIVQ, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.430284135 (512h50m49.30136717s) [noise=true , address=145.79.227.49:4338 , piece_id=CZSRF4DOJTWCEF6QPFGHKGUEOMHVQWHUZFNK2QW6PH4OYSXZQ7BA, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.430698854 (512h50m49.301781889s) [noise=true , address=145.79.227.45:4230 , piece_id=P46DAMMUTTWYBLINQKQGX7VOFXHSELMNM274DWXGB56VOD3XAVKA, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:49.431117481 (512h50m49.302200526s) [noise=true , address=77.90.31.14:4664 , piece_id=Z3GHZLOAAP55WRNNXEKVNLAP7CL2P2VIVXXWHHK2PEUBM4JSQQWQ, seg_pos={0 0}, upload=450021]: Uploading piece... Dec 27 00:19:51.052320756 (512h50m50.923403801s) [upload=450021]: Next returned nil segment Dec 27 00:19:51.052341987 (512h50m50.923425032s) [upload=450021]: Waiting for error group managing segments... Dec 27 00:19:51.432705625 (512h50m51.30378866s) [noise=true , address=145.79.227.2:4114 , piece_id=HRBXCRTY4F2KQYPWDRYDCRV3JIS25FGKPUHE23QRUKH7OY74MWQA, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.436100890 (512h50m51.307183935s) [noise=true , address=145.79.227.50:4373 , piece_id=M42YYUSUI2XL7LBMI5RT65VOWYIYFC7XFE6JFKPMWP33ESSMXXMQ, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.438592394 (512h50m51.309675449s) [noise=true , address=145.79.227.13:4804 , piece_id=JF2I4CDZ737IDWIDA2D43NQLHBE6C634HMCVOBRMS4MKKS4LB73Q, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.439851842 (512h50m51.310934877s) [noise=true , address=145.79.227.46:4602 , piece_id=R7LUXLSCEY4CZWSX3Y2DE7BTMMKFISLFLVJERCPF7VD5OEIXAENQ, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.443349853 (512h50m51.314432898s) [noise=true , address=145.79.227.38:4978 , piece_id=UE5ZHWVK2CCNX7MBF7PLQ4VRGTXWW2VZYWUE2T32XMGGZ3HDB4XA, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.443428412 (512h50m51.314511447s) [noise=true , address=145.79.227.12:4765 , piece_id=KRGHSNDXMBJAXQJDJPRGC2CECBY472ONWHEG7WTMW3YD55A3AFFQ, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.448193225 (512h50m51.31927626s) [noise=true , address=145.79.227.42:4122 , piece_id=PCDQN2P5EMM2PRUKHXBNMJFWCRQLQPBILUSQPE4JUXRHCK7UX3YA, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.458696645 (512h50m51.32977969s) [noise=true , address=145.79.227.36:4518 , piece_id=F6TIEVNRV7265TAKZHBV3AA3TUYQKU5G4W2MUCOOK5275XTSXCVQ, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.465233381 (512h50m51.336316426s) [noise=true , address=145.79.227.49:4338 , piece_id=CZSRF4DOJTWCEF6QPFGHKGUEOMHVQWHUZFNK2QW6PH4OYSXZQ7BA, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.472526638 (512h50m51.343609684s) [noise=true , address=145.79.227.45:4230 , piece_id=P46DAMMUTTWYBLINQKQGX7VOFXHSELMNM274DWXGB56VOD3XAVKA, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.639710928 (512h50m51.510793963s) [noise=true , address=77.90.31.14:4664 , piece_id=Z3GHZLOAAP55WRNNXEKVNLAP7CL2P2VIVXXWHHK2PEUBM4JSQQWQ, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.641425442 (512h50m51.512508477s) [noise=true , address=45.91.116.8:4045 , piece_id=KJ3D3HTVI4L5GXWYXEKKDTBCXADH6JCGVOIYPLFBCHT3H5FZRSKA, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.650862320 (512h50m51.521945365s) [noise=true , address=77.90.31.17:4773 , piece_id=GNJNZSSRMWGGFJ3LVAONWNTULVO6X4OQW3NKHVSC7PC76RPEZ47Q, seg_pos={0 0}, upload=450021]: Done uploading piece. err: <nil> Dec 27 00:19:51.650886767 (512h50m51.521969812s) [seg_pos={0 0}, upload=450021]: Segment reached optimal threshold of 13 pieces. Dec 27 00:19:51.651132435 (512h50m51.52221547s) [noise=true , address=77.90.31.10:4519 , piece_id=X6ALESIWGNAFPYETMJMQY6NGBFTBSBOSDSCJNGX75POL2GL6SBSA, seg_pos={0 0}, upload=450021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 00:19:51.651140270 (512h50m51.522223305s) [noise=true , address=77.90.31.10:4519 , piece_id=X6ALESIWGNAFPYETMJMQY6NGBFTBSBOSDSCJNGX75POL2GL6SBSA, seg_pos={0 0}, upload=450021]: Long tail cancellation due to optimal threshold Dec 27 00:19:51.651286920 (512h50m51.522369965s) [noise=true , address=45.91.116.14:4189 , piece_id=X3TU6TUQ5AOKLJMBA3BX7EBJ5YANCIRWFXN7MOAU26MV3XHD242Q, seg_pos={0 0}, upload=450021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 00:19:51.651304322 (512h50m51.522387357s) [noise=true , address=45.91.116.14:4189 , piece_id=X3TU6TUQ5AOKLJMBA3BX7EBJ5YANCIRWFXN7MOAU26MV3XHD242Q, seg_pos={0 0}, upload=450021]: Long tail cancellation due to optimal threshold Dec 27 00:19:51.651382471 (512h50m51.522465506s) [noise=true , address=45.91.116.11:4118 , piece_id=MRENDTVHTQU77N2OKS2LL6TRU2IZGPASCCIJI2BUYBS4LD33CIVQ, seg_pos={0 0}, upload=450021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:59074->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:59074->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59074->45.91.116.11:4118: use of closed network connection Dec 27 00:19:51.651389143 (512h50m51.522472188s) [noise=true , address=45.91.116.11:4118 , piece_id=MRENDTVHTQU77N2OKS2LL6TRU2IZGPASCCIJI2BUYBS4LD33CIVQ, seg_pos={0 0}, upload=450021]: Long tail cancellation due to optimal threshold Dec 27 00:19:51.651404273 (512h50m51.522487318s) [seg_pos={0 0}, upload=450021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:19:51.651476671 (512h50m51.522559716s) [upload=450021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:19:51.651481039 (512h50m51.522564074s) [upload=450021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:19:51.950939112 (512h50m51.822022147s) [upload=450021]: Done uploading segments... Dec 27 00:19:51.950956987 (512h50m51.822040022s) [upload=450021]: Upload finished. err: <nil> Duration: 30.003017667s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 244.433471ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:19:57 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 244.456646ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 249.330575ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:19:17 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 249.359219ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XX.XX..XXXX..X.XX...X.........X.........X.......X.XX.XX.XX] duration: 30.003767117s trace: https://jaeger.dev.storj.io/trace/4cb9ec3286c7bec9 FAIL (2025-12-27 00:19:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 49m36.740000566s) [upload=450006]: Done starting upload Dec 27 00:18:36.869002743 (512h49m36.740085788s) [upload=450006]: Uploading segments... Dec 27 00:18:36.869103735 (512h49m36.74018677s) [upload=450006]: Got next segment. Inline: false Dec 27 00:18:36.869134443 (512h49m36.740217478s) [upload=450006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:18:36.869138531 (512h49m36.740221566s) [upload=450006]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:18:37.114234403 (512h49m36.985317438s) [seg_pos={0 0}, upload=450006]: Begin upload segment... Dec 27 00:18:37.114292624 (512h49m36.985375659s) [seg_pos={0 0}, upload=450006]: Done begin upload segment. Dec 27 00:18:37.114316600 (512h49m36.985399635s) [noise=true , address=145.79.227.46:4603 , piece_id=FS2UV2HVZ45L6I6L5A2LGQO2SOVQIKIGYWUT4RLUC637NSWPE7JA, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.114326539 (512h49m36.985409574s) [noise=true , address=145.79.227.13:4804 , piece_id=6GLMIMYDQPX2P35SXT2TAJPNBJYA2LCCLJLU6ZWOYUWMWB3EBBFA, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.114373388 (512h49m36.985456423s) [noise=true , address=145.79.227.6:4257 , piece_id=HC7VCFCEPD2YHWZZUJVCLGUIJ5ETUPD4F55TZ5OFCYELKXQW3XTQ, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.114874863 (512h49m36.985957898s) [noise=true , address=145.79.227.14:4837 , piece_id=BEGVANQL57JJM5EF2LJDYHIVV52KHI7FEPAYGNHRVHDL4DGTZTEA, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.115142162 (512h49m36.986225197s) [noise=true , address=145.79.227.38:4977 , piece_id=2SOSBUV23KQAPKNUIOQEHJ37T45JGQ5GDQBF7F6FOM4VO7NYW53A, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.115188050 (512h49m36.986271085s) [noise=true , address=145.79.227.17:4972 , piece_id=I5LW37W6SUXC5THU74PD5OWBXLCXDJCD22WCLELKCX6EXEXUZW7A, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.115785968 (512h49m36.986869003s) [noise=true , address=145.79.227.19:4079 , piece_id=OCEZL72JFMUSU3DJRPI4H4HKTAIXZOEVHFP54GZJBXMEIJDJUUKA, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.116071792 (512h49m36.987154837s) [noise=true , address=145.79.227.37:4567 , piece_id=GUZ2SXDTVHYVAZTQ33P4EENJ4SD5FBR5BVTBSJM344O3YPQA3TKQ, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.116335904 (512h49m36.987418939s) [noise=true , address=77.90.31.14:4661 , piece_id=CU74O3HRQPMQE5JUGMJU2JYPRKKSWKKL6RYJVLY7XQSPLWSJSP4A, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.116649862 (512h49m36.987732897s) [noise=true , address=45.91.116.13:4155 , piece_id=2QT4LHW6ACTRXOKC7GM3OSC3EBJJHWXHGCLSB6T7A5ZEFZXOWKWA, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.116792464 (512h49m36.987875499s) [noise=true , address=77.90.31.13:4629 , piece_id=J45I72JRMZU4LYYILROGZU6C7ZLIME5XGT2ZWQ642VOV2JR6SCMQ, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.117180734 (512h49m36.988263769s) [noise=true , address=45.91.116.6:4976 , piece_id=HWEK5TNUOHNY6UJBGSWDE45OCLAZ2SV4T3VGYITOH3KDOORP6TSA, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.117612075 (512h49m36.98869512s) [noise=true , address=145.79.227.50:4373 , piece_id=J3HFU3TJSB5ESEQAER3C7X5GVC6F7S2UGSFQ7XE5TYI3OFDK3PNA, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.117832324 (512h49m36.988915359s) [noise=true , address=45.91.116.4:4939 , piece_id=Q3XS2NSDLTBZRONVA5KAGCX6SN5NAL3OJKCM6SNAGCHIIT556KLQ, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.118206717 (512h49m36.989289752s) [noise=true , address=145.79.227.11:4730 , piece_id=CD56OZXNEUCVDGCDMG4MW2TH65LFD2COKMFXSCMJFCT5QSZGJ57A, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:37.118556122 (512h49m36.989639157s) [noise=true , address=145.79.227.48:4303 , piece_id=REP6TA35IOG6RYEUAJOIPSMHCWOSFM54TOP2NCTDV4LVEXD5MF5A, seg_pos={0 0}, upload=450006]: Uploading piece... Dec 27 00:18:38.956546796 (512h49m38.827629831s) [upload=450006]: Next returned nil segment Dec 27 00:18:38.956568607 (512h49m38.827651642s) [upload=450006]: Waiting for error group managing segments... Dec 27 00:18:39.265529587 (512h49m39.136612632s) [noise=true , address=145.79.227.37:4567 , piece_id=GUZ2SXDTVHYVAZTQ33P4EENJ4SD5FBR5BVTBSJM344O3YPQA3TKQ, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.282021459 (512h49m39.153104494s) [noise=true , address=145.79.227.11:4730 , piece_id=CD56OZXNEUCVDGCDMG4MW2TH65LFD2COKMFXSCMJFCT5QSZGJ57A, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.282327602 (512h49m39.153410647s) [noise=true , address=145.79.227.13:4804 , piece_id=6GLMIMYDQPX2P35SXT2TAJPNBJYA2LCCLJLU6ZWOYUWMWB3EBBFA, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.296363578 (512h49m39.167446613s) [noise=true , address=145.79.227.48:4303 , piece_id=REP6TA35IOG6RYEUAJOIPSMHCWOSFM54TOP2NCTDV4LVEXD5MF5A, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.303367555 (512h49m39.1744506s) [noise=true , address=145.79.227.46:4603 , piece_id=FS2UV2HVZ45L6I6L5A2LGQO2SOVQIKIGYWUT4RLUC637NSWPE7JA, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.306298856 (512h49m39.177381892s) [noise=true , address=145.79.227.50:4373 , piece_id=J3HFU3TJSB5ESEQAER3C7X5GVC6F7S2UGSFQ7XE5TYI3OFDK3PNA, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.309138463 (512h49m39.180221498s) [noise=true , address=145.79.227.19:4079 , piece_id=OCEZL72JFMUSU3DJRPI4H4HKTAIXZOEVHFP54GZJBXMEIJDJUUKA, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.316744586 (512h49m39.187827621s) [noise=true , address=145.79.227.17:4972 , piece_id=I5LW37W6SUXC5THU74PD5OWBXLCXDJCD22WCLELKCX6EXEXUZW7A, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.345776305 (512h49m39.21685934s) [noise=true , address=45.91.116.4:4939 , piece_id=Q3XS2NSDLTBZRONVA5KAGCX6SN5NAL3OJKCM6SNAGCHIIT556KLQ, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.352291601 (512h49m39.223374646s) [noise=true , address=77.90.31.14:4661 , piece_id=CU74O3HRQPMQE5JUGMJU2JYPRKKSWKKL6RYJVLY7XQSPLWSJSP4A, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.353541942 (512h49m39.224624977s) [noise=true , address=77.90.31.13:4629 , piece_id=J45I72JRMZU4LYYILROGZU6C7ZLIME5XGT2ZWQ642VOV2JR6SCMQ, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.414074624 (512h49m39.285157669s) [noise=true , address=145.79.227.14:4837 , piece_id=BEGVANQL57JJM5EF2LJDYHIVV52KHI7FEPAYGNHRVHDL4DGTZTEA, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.429525024 (512h49m39.300608069s) [noise=true , address=45.91.116.13:4155 , piece_id=2QT4LHW6ACTRXOKC7GM3OSC3EBJJHWXHGCLSB6T7A5ZEFZXOWKWA, seg_pos={0 0}, upload=450006]: Done uploading piece. err: <nil> Dec 27 00:18:39.429548578 (512h49m39.300631613s) [seg_pos={0 0}, upload=450006]: Segment reached optimal threshold of 13 pieces. Dec 27 00:18:39.429774218 (512h49m39.300857253s) [noise=true , address=45.91.116.6:4976 , piece_id=HWEK5TNUOHNY6UJBGSWDE45OCLAZ2SV4T3VGYITOH3KDOORP6TSA, seg_pos={0 0}, upload=450006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 00:18:39.429783796 (512h49m39.300866831s) [noise=true , address=45.91.116.6:4976 , piece_id=HWEK5TNUOHNY6UJBGSWDE45OCLAZ2SV4T3VGYITOH3KDOORP6TSA, seg_pos={0 0}, upload=450006]: Long tail cancellation due to optimal threshold Dec 27 00:18:39.430039714 (512h49m39.301122759s) [noise=true , address=145.79.227.38:4977 , piece_id=2SOSBUV23KQAPKNUIOQEHJ37T45JGQ5GDQBF7F6FOM4VO7NYW53A, seg_pos={0 0}, upload=450006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:38520->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:38520->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38520->145.79.227.38:4977: use of closed network connection Dec 27 00:18:39.430058379 (512h49m39.301141424s) [noise=true , address=145.79.227.38:4977 , piece_id=2SOSBUV23KQAPKNUIOQEHJ37T45JGQ5GDQBF7F6FOM4VO7NYW53A, seg_pos={0 0}, upload=450006]: Long tail cancellation due to optimal threshold Dec 27 00:18:39.430224065 (512h49m39.30130711s) [noise=true , address=145.79.227.6:4257 , piece_id=HC7VCFCEPD2YHWZZUJVCLGUIJ5ETUPD4F55TZ5OFCYELKXQW3XTQ, seg_pos={0 0}, upload=450006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: read tcp 188.166.225.170:53292->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 00:18:39.430264762 (512h49m39.301347797s) [noise=true , address=145.79.227.6:4257 , piece_id=HC7VCFCEPD2YHWZZUJVCLGUIJ5ETUPD4F55TZ5OFCYELKXQW3XTQ, seg_pos={0 0}, upload=450006]: Long tail cancellation due to optimal threshold Dec 27 00:18:39.430288357 (512h49m39.301371402s) [seg_pos={0 0}, upload=450006]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:18:39.430377667 (512h49m39.301460712s) [upload=450006]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:18:39.430384069 (512h49m39.301467114s) [upload=450006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:18:39.724884825 (512h49m39.59596786s) [upload=450006]: Done uploading segments... Dec 27 00:18:39.724901196 (512h49m39.595984241s) [upload=450006]: Upload finished. err: <nil> Duration: 30.00378932s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 240.448682ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:18:21 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 240.480813ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..XX.XX..XXXX..X.XX...X.........X.........X.......X.XX.XX.X] duration: 30.003530539s trace: https://jaeger.dev.storj.io/trace/3967cb08e988229b FAIL (2025-12-27 00:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:17:48.408042680 (512h48m48.279125715s) [upload=449997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:17:48.653922785 (512h48m48.52500582s) [seg_pos={0 0}, upload=449997]: Begin upload segment... Dec 27 00:17:48.653978752 (512h48m48.525061787s) [seg_pos={0 0}, upload=449997]: Done begin upload segment. Dec 27 00:17:48.654011024 (512h48m48.525094059s) [noise=true , address=145.79.227.49:4337 , piece_id=WKAZX3C5CFTCMYIAO4V27ZPQA64XYADTPFG7FXBM6WVO47HZ3N2Q, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.654176419 (512h48m48.525259454s) [noise=true , address=45.91.116.8:4048 , piece_id=3G23SZE4TNMDLHJYMKGQGAH2XDWQWALIJ67K5G4GJBLVPAJT7LMA, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.654654378 (512h48m48.525737413s) [noise=true , address=145.79.227.50:4375 , piece_id=CVWNDPT2DWZ5OXRHQTNYC5FAZZI5AUVBZD66YL3SSMOGUTKT4HUA, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.654942517 (512h48m48.526025553s) [noise=true , address=145.79.227.2:4113 , piece_id=IWQ73WRIOJO54PLS42FGQK5QEYOJMNMHDBMFEZ2EZ6HKVNKAGWJQ, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.655211489 (512h48m48.526294524s) [noise=true , address=45.91.116.7:4010 , piece_id=5XANVU35VBEHBBADQMLBINPIT4KIPT3ERLPIZFHQQQR26VXLJFEQ, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.655487965 (512h48m48.526571011s) [noise=true , address=145.79.227.39:4014 , piece_id=GYXB2XB4IED3DVIUI7MRSQDUO6EINUJ3J4J4U5ZBG5UB2J5IIKVA, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.655753021 (512h48m48.526836056s) [noise=true , address=145.79.227.35:4481 , piece_id=OUTODPEP6ZSA3SO5GPBJ5E5LQN2REYFVJLLUCRPOQPBASV6IACVQ, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.656072188 (512h48m48.527155233s) [noise=true , address=77.90.31.15:4697 , piece_id=JAEV6MUSVRP4F54SFY3H4PEMA3E5OB33JPL3GK6KKTS7IBPHUQ6A, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.657562666 (512h48m48.528645701s) [noise=true , address=145.79.227.46:4603 , piece_id=5AFR2MGKZDUXBUXMROAH2Q6WKZKZJWYUA6NZ7IZH25SXOBSPNE5Q, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.658749075 (512h48m48.529832111s) [noise=true , address=77.90.31.10:4519 , piece_id=3O34I6TC7Y7GGWX77LREGY3PJ52GPGNNV3N5S6FFRNA4OKDUSMWA, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.659884518 (512h48m48.530967553s) [noise=true , address=45.91.116.6:4975 , piece_id=ZVWYUGNGVTFROHXP636AWTUACM5JR7DAG3WQQDVNFHI6GHHKGQ5A, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.660907866 (512h48m48.531991112s) [noise=true , address=77.90.31.11:4555 , piece_id=SB3JK5Z2EGQ2A2PMFEY743V7ES5MSFHCO4DY4RY5IG4MORVELENQ, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.662237117 (512h48m48.533320162s) [noise=true , address=145.79.227.16:4911 , piece_id=IWNB6M3PEMX645NXXFRT3N76PBNCNUGKLJLIIUAXKGFXFDAV4GJA, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.664302950 (512h48m48.535385986s) [noise=true , address=45.91.116.4:4939 , piece_id=SKSRQQIETG3FSNYZTMCQNRW5CH6L3HU2AYLBLSIEUQECNHCH4XDA, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.665495812 (512h48m48.536578847s) [noise=true , address=45.91.116.12:4228 , piece_id=ZKYXDJRU35W6HECY35NRIF2AHSUUQRZL6MNBVULV5QKTZ5TCY35A, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:48.666815344 (512h48m48.537898389s) [noise=true , address=77.90.31.14:4662 , piece_id=G2P56ILCFVRDGV5IVOGB6FTYZPZXFYG7R7Q6TFA26VBJZCGM3WUA, seg_pos={0 0}, upload=449997]: Uploading piece... Dec 27 00:17:50.356597583 (512h48m50.227680628s) [upload=449997]: Next returned nil segment Dec 27 00:17:50.356617931 (512h48m50.227700966s) [upload=449997]: Waiting for error group managing segments... Dec 27 00:17:50.642682302 (512h48m50.513765337s) [noise=true , address=145.79.227.50:4375 , piece_id=CVWNDPT2DWZ5OXRHQTNYC5FAZZI5AUVBZD66YL3SSMOGUTKT4HUA, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:50.652977425 (512h48m50.52406047s) [noise=true , address=145.79.227.2:4113 , piece_id=IWQ73WRIOJO54PLS42FGQK5QEYOJMNMHDBMFEZ2EZ6HKVNKAGWJQ, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:50.675162207 (512h48m50.546245252s) [noise=true , address=145.79.227.16:4911 , piece_id=IWNB6M3PEMX645NXXFRT3N76PBNCNUGKLJLIIUAXKGFXFDAV4GJA, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:50.675354694 (512h48m50.546437729s) [noise=true , address=145.79.227.39:4014 , piece_id=GYXB2XB4IED3DVIUI7MRSQDUO6EINUJ3J4J4U5ZBG5UB2J5IIKVA, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:50.679686252 (512h48m50.550769287s) [noise=true , address=145.79.227.49:4337 , piece_id=WKAZX3C5CFTCMYIAO4V27ZPQA64XYADTPFG7FXBM6WVO47HZ3N2Q, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:50.682054491 (512h48m50.553137527s) [noise=true , address=145.79.227.35:4481 , piece_id=OUTODPEP6ZSA3SO5GPBJ5E5LQN2REYFVJLLUCRPOQPBASV6IACVQ, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:50.698997764 (512h48m50.570080809s) [noise=true , address=145.79.227.46:4603 , piece_id=5AFR2MGKZDUXBUXMROAH2Q6WKZKZJWYUA6NZ7IZH25SXOBSPNE5Q, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:50.824542376 (512h48m50.695625411s) [noise=true , address=77.90.31.10:4519 , piece_id=3O34I6TC7Y7GGWX77LREGY3PJ52GPGNNV3N5S6FFRNA4OKDUSMWA, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:50.886677879 (512h48m50.757760914s) [noise=true , address=77.90.31.15:4697 , piece_id=JAEV6MUSVRP4F54SFY3H4PEMA3E5OB33JPL3GK6KKTS7IBPHUQ6A, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:50.991168231 (512h48m50.862251276s) [noise=true , address=45.91.116.8:4048 , piece_id=3G23SZE4TNMDLHJYMKGQGAH2XDWQWALIJ67K5G4GJBLVPAJT7LMA, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:51.011009601 (512h48m50.882092646s) [noise=true , address=45.91.116.12:4228 , piece_id=ZKYXDJRU35W6HECY35NRIF2AHSUUQRZL6MNBVULV5QKTZ5TCY35A, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:51.043279297 (512h48m50.914362342s) [noise=true , address=45.91.116.4:4939 , piece_id=SKSRQQIETG3FSNYZTMCQNRW5CH6L3HU2AYLBLSIEUQECNHCH4XDA, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:51.061133163 (512h48m50.932216208s) [noise=true , address=77.90.31.11:4555 , piece_id=SB3JK5Z2EGQ2A2PMFEY743V7ES5MSFHCO4DY4RY5IG4MORVELENQ, seg_pos={0 0}, upload=449997]: Done uploading piece. err: <nil> Dec 27 00:17:51.061253712 (512h48m50.932336747s) [seg_pos={0 0}, upload=449997]: Segment reached optimal threshold of 13 pieces. Dec 27 00:17:51.061434366 (512h48m50.932517401s) [noise=true , address=45.91.116.7:4010 , piece_id=5XANVU35VBEHBBADQMLBINPIT4KIPT3ERLPIZFHQQQR26VXLJFEQ, seg_pos={0 0}, upload=449997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:49394->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:49394->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49394->45.91.116.7:4010: use of closed network connection Dec 27 00:17:51.061440527 (512h48m50.932523573s) [noise=true , address=45.91.116.7:4010 , piece_id=5XANVU35VBEHBBADQMLBINPIT4KIPT3ERLPIZFHQQQR26VXLJFEQ, seg_pos={0 0}, upload=449997]: Long tail cancellation due to optimal threshold Dec 27 00:17:51.061458342 (512h48m50.932541377s) [noise=true , address=45.91.116.6:4975 , piece_id=ZVWYUGNGVTFROHXP636AWTUACM5JR7DAG3WQQDVNFHI6GHHKGQ5A, seg_pos={0 0}, upload=449997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:55420->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:55420->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55420->45.91.116.6:4975: use of closed network connection Dec 27 00:17:51.061466728 (512h48m50.932549763s) [noise=true , address=45.91.116.6:4975 , piece_id=ZVWYUGNGVTFROHXP636AWTUACM5JR7DAG3WQQDVNFHI6GHHKGQ5A, seg_pos={0 0}, upload=449997]: Long tail cancellation due to optimal threshold Dec 27 00:17:51.061682338 (512h48m50.932765383s) [noise=true , address=77.90.31.14:4662 , piece_id=G2P56ILCFVRDGV5IVOGB6FTYZPZXFYG7R7Q6TFA26VBJZCGM3WUA, seg_pos={0 0}, upload=449997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:40172->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:40172->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: context canceled Dec 27 00:17:51.061690434 (512h48m50.932773499s) [noise=true , address=77.90.31.14:4662 , piece_id=G2P56ILCFVRDGV5IVOGB6FTYZPZXFYG7R7Q6TFA26VBJZCGM3WUA, seg_pos={0 0}, upload=449997]: Long tail cancellation due to optimal threshold Dec 27 00:17:51.061709149 (512h48m50.932792194s) [seg_pos={0 0}, upload=449997]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:17:51.061789963 (512h48m50.932872998s) [upload=449997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:17:51.061821623 (512h48m50.932904658s) [upload=449997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:17:51.356386190 (512h48m51.227469225s) [upload=449997]: Done uploading segments... Dec 27 00:17:51.356441345 (512h48m51.22752439s) [upload=449997]: Upload finished. err: <nil> Duration: 30.003557219s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 703.598535ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:18:00 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 703.623993ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 254.935899ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:17:25 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 254.967247ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..XX.XX..XXXX..X.XX...X.........X.........X.......X.XX.XX.] duration: 6.011588172s trace: https://jaeger.dev.storj.io/trace/7ede96b7d006e1df FAIL (2025-12-27 00:16:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: EBZCOPRTTJG4WOQALJHIW7EZK5SZBHJDKSSXDDTZHD23AHYOHPCQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: PLVVBC3DBKNXMO6MSWPWAGAIWHGXJFVTY2K2EB6MV5MYXXFVNC4Q) context deadline exceeded ======LOGS====== Dec 27 00:15:58.903056615 (512h46m58.77413966s) [upload=449972]: Starting upload Dec 27 00:15:58.903131126 (512h46m58.774214171s) [upload=449972]: Done starting upload Dec 27 00:15:58.903292814 (512h46m58.774375849s) [upload=449972]: Uploading segments... Dec 27 00:15:58.903373358 (512h46m58.774456403s) [upload=449972]: Got next segment. Inline: false Dec 27 00:15:58.903558851 (512h46m58.774641896s) [upload=449972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:15:58.903566315 (512h46m58.77464936s) [upload=449972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:15:59.147213668 (512h46m59.018296713s) [seg_pos={0 0}, upload=449972]: Begin upload segment... Dec 27 00:15:59.147265085 (512h46m59.01834813s) [seg_pos={0 0}, upload=449972]: Done begin upload segment. Dec 27 00:15:59.147295914 (512h46m59.018378959s) [noise=true , address=145.79.227.46:4601 , piece_id=FW4SHC75P33CDXR2TYFWGIHOBFINYWL4B5B5WX5URMNFQYMSZH4A, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.147319159 (512h46m59.018402204s) [noise=true , address=45.91.116.6:4975 , piece_id=EBZCOPRTTJG4WOQALJHIW7EZK5SZBHJDKSSXDDTZHD23AHYOHPCQ, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.147888392 (512h46m59.018971437s) [noise=true , address=45.91.116.9:4264 , piece_id=CUK3YC72WCBNK5S2C4A4CRQWA7ISDMFXONVW3E4HXIDJCPT2NP6A, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.148346334 (512h46m59.019429379s) [noise=true , address=145.79.227.5:4222 , piece_id=AEH2PTPLHIZUEJV635KQ6GBJA33REYU4EUHDYK75BNSUBCWWM5RQ, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.148817141 (512h46m59.019900186s) [noise=true , address=145.79.227.47:4265 , piece_id=MAKLCUQ7GV2SGLLJE73EEFSFPUD4HXDFPQKVUIKKTMHY25FJTE3Q, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.149293538 (512h46m59.020376573s) [noise=true , address=145.79.227.3:4151 , piece_id=B7JH6YDCAOQNIFG42IGEAW3JWVX4DRB2PO2UNA5R7YNBYUKVKTTQ, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.149931824 (512h46m59.021014859s) [noise=true , address=145.79.227.36:4518 , piece_id=DE3O4GUAHMQ25WIBUT4PVH2L427KD2QC4362ZJHJ4S2AOPSLDMTA, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.150533028 (512h46m59.021616073s) [noise=true , address=145.79.227.50:4375 , piece_id=J6TMBQDMVCA7SRHMIH526LVPCCTDOADECOI3QJDKA75UTSOGLQ4A, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.150550661 (512h46m59.021633716s) [noise=true , address=145.79.227.14:4837 , piece_id=2UGRZP6VJMORNORY6NXXECZKCE3XUUX4IT6GS2TIFBPVJ7SOE5TQ, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.151068207 (512h46m59.022151242s) [noise=true , address=77.90.31.15:4700 , piece_id=CV5PAC7KCIHUJSHH6R3W3YQ53RMK4YUIF2HWYMPHP2PK5P67Q6AA, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.151599478 (512h46m59.022682513s) [noise=true , address=145.79.227.42:4123 , piece_id=K3GYGSH6WAAOYFUNGKG3ZJLBUTHAEWWX3JHXPQKZDAOW2FD44Q4Q, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.152230720 (512h46m59.023313796s) [noise=true , address=145.79.227.37:4567 , piece_id=TBPFIBTMLDFKTJ4LNKSUBZ6MTR3WH4BWMMQM77U3EOVO4SZW5I5A, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.153146635 (512h46m59.02422967s) [noise=true , address=45.91.116.5:4297 , piece_id=PLVVBC3DBKNXMO6MSWPWAGAIWHGXJFVTY2K2EB6MV5MYXXFVNC4Q, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.154944888 (512h46m59.026027944s) [noise=true , address=45.91.116.11:4119 , piece_id=PDFKHLRFJTTAXUU4R7FL3VRAAH7QWRBGSJIHAT7KKHBOORMDCHHA, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.156795182 (512h46m59.027878227s) [noise=true , address=145.79.227.17:4971 , piece_id=NAFDG2U3MMWFMY7USYNVNYU5VP3QFT3YVADEJ2O7VHUCKSF3IRTA, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.169324770 (512h46m59.040407875s) [noise=true , address=145.79.227.48:4302 , piece_id=YDJ63ZQF45A2JCCKEMTXCR3IJWK3WX7C2GM2EMLI7JP7LO4MAOFA, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:16:00.761283060 (512h47m0.632366105s) [upload=449972]: Next returned nil segment Dec 27 00:16:00.761298759 (512h47m0.632381804s) [upload=449972]: Waiting for error group managing segments... Dec 27 00:16:01.147955290 (512h47m1.019038335s) [noise=true , address=145.79.227.14:4837 , piece_id=2UGRZP6VJMORNORY6NXXECZKCE3XUUX4IT6GS2TIFBPVJ7SOE5TQ, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.148866495 (512h47m1.01994953s) [noise=true , address=145.79.227.47:4265 , piece_id=MAKLCUQ7GV2SGLLJE73EEFSFPUD4HXDFPQKVUIKKTMHY25FJTE3Q, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.155582503 (512h47m1.026665538s) [noise=true , address=145.79.227.5:4222 , piece_id=AEH2PTPLHIZUEJV635KQ6GBJA33REYU4EUHDYK75BNSUBCWWM5RQ, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.159391706 (512h47m1.030474751s) [noise=true , address=145.79.227.50:4375 , piece_id=J6TMBQDMVCA7SRHMIH526LVPCCTDOADECOI3QJDKA75UTSOGLQ4A, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.159434457 (512h47m1.030517502s) [noise=true , address=145.79.227.46:4601 , piece_id=FW4SHC75P33CDXR2TYFWGIHOBFINYWL4B5B5WX5URMNFQYMSZH4A, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.169768094 (512h47m1.040851129s) [noise=true , address=145.79.227.48:4302 , piece_id=YDJ63ZQF45A2JCCKEMTXCR3IJWK3WX7C2GM2EMLI7JP7LO4MAOFA, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.182193884 (512h47m1.053276929s) [noise=true , address=145.79.227.42:4123 , piece_id=K3GYGSH6WAAOYFUNGKG3ZJLBUTHAEWWX3JHXPQKZDAOW2FD44Q4Q, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.187378066 (512h47m1.058461111s) [noise=true , address=145.79.227.36:4518 , piece_id=DE3O4GUAHMQ25WIBUT4PVH2L427KD2QC4362ZJHJ4S2AOPSLDMTA, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.190972200 (512h47m1.062055235s) [noise=true , address=145.79.227.3:4151 , piece_id=B7JH6YDCAOQNIFG42IGEAW3JWVX4DRB2PO2UNA5R7YNBYUKVKTTQ, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.192681855 (512h47m1.06376489s) [noise=true , address=145.79.227.37:4567 , piece_id=TBPFIBTMLDFKTJ4LNKSUBZ6MTR3WH4BWMMQM77U3EOVO4SZW5I5A, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.195930270 (512h47m1.067013315s) [noise=true , address=145.79.227.17:4971 , piece_id=NAFDG2U3MMWFMY7USYNVNYU5VP3QFT3YVADEJ2O7VHUCKSF3IRTA, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.394477788 (512h47m1.265560833s) [noise=true , address=45.91.116.5:4297 , piece_id=PLVVBC3DBKNXMO6MSWPWAGAIWHGXJFVTY2K2EB6MV5MYXXFVNC4Q, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.430641829 (512h47m1.301724874s) [noise=true , address=45.91.116.6:4975 , piece_id=EBZCOPRTTJG4WOQALJHIW7EZK5SZBHJDKSSXDDTZHD23AHYOHPCQ, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.430712834 (512h47m1.301795889s) [seg_pos={0 0}, upload=449972]: Segment reached optimal threshold of 13 pieces. Dec 27 00:16:01.430955266 (512h47m1.302038301s) [noise=true , address=45.91.116.9:4264 , piece_id=CUK3YC72WCBNK5S2C4A4CRQWA7ISDMFXONVW3E4HXIDJCPT2NP6A, seg_pos={0 0}, upload=449972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 27 00:16:01.430967218 (512h47m1.302050253s) [noise=true , address=45.91.116.9:4264 , piece_id=CUK3YC72WCBNK5S2C4A4CRQWA7ISDMFXONVW3E4HXIDJCPT2NP6A, seg_pos={0 0}, upload=449972]: Long tail cancellation due to optimal threshold Dec 27 00:16:01.431218727 (512h47m1.302301762s) [noise=true , address=77.90.31.15:4700 , piece_id=CV5PAC7KCIHUJSHH6R3W3YQ53RMK4YUIF2HWYMPHP2PK5P67Q6AA, seg_pos={0 0}, upload=449972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 00:16:01.431225751 (512h47m1.302308786s) [noise=true , address=77.90.31.15:4700 , piece_id=CV5PAC7KCIHUJSHH6R3W3YQ53RMK4YUIF2HWYMPHP2PK5P67Q6AA, seg_pos={0 0}, upload=449972]: Long tail cancellation due to optimal threshold Dec 27 00:16:01.431463153 (512h47m1.302546189s) [noise=true , address=45.91.116.11:4119 , piece_id=PDFKHLRFJTTAXUU4R7FL3VRAAH7QWRBGSJIHAT7KKHBOORMDCHHA, seg_pos={0 0}, upload=449972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 27 00:16:01.431484123 (512h47m1.302567188s) [noise=true , address=45.91.116.11:4119 , piece_id=PDFKHLRFJTTAXUU4R7FL3VRAAH7QWRBGSJIHAT7KKHBOORMDCHHA, seg_pos={0 0}, upload=449972]: Long tail cancellation due to optimal threshold Dec 27 00:16:01.431520432 (512h47m1.302603477s) [seg_pos={0 0}, upload=449972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:16:01.431647144 (512h47m1.302730189s) [upload=449972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:16:01.431656942 (512h47m1.302739987s) [upload=449972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:16:01.713942788 (512h47m1.585025833s) [upload=449972]: Done uploading segments... Dec 27 00:16:01.713968075 (512h47m1.58505112s) [upload=449972]: Upload finished. err: <nil> Duration: 6.011615345s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 261.948021ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:16:50 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 261.967969ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X..XX.XX..XXXX..X.XX...X.........X.........X.......X.XX.XX] duration: 30.002146802s trace: https://jaeger.dev.storj.io/trace/7ede96b7d006e1df FAIL (2025-12-27 00:16:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: EBZCOPRTTJG4WOQALJHIW7EZK5SZBHJDKSSXDDTZHD23AHYOHPCQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: PLVVBC3DBKNXMO6MSWPWAGAIWHGXJFVTY2K2EB6MV5MYXXFVNC4Q) context deadline exceeded ======LOGS====== Dec 27 00:15:58.903056615 (512h46m58.77413966s) [upload=449972]: Starting upload Dec 27 00:15:58.903131126 (512h46m58.774214171s) [upload=449972]: Done starting upload Dec 27 00:15:58.903292814 (512h46m58.774375849s) [upload=449972]: Uploading segments... Dec 27 00:15:58.903373358 (512h46m58.774456403s) [upload=449972]: Got next segment. Inline: false Dec 27 00:15:58.903558851 (512h46m58.774641896s) [upload=449972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:15:58.903566315 (512h46m58.77464936s) [upload=449972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:15:59.147213668 (512h46m59.018296713s) [seg_pos={0 0}, upload=449972]: Begin upload segment... Dec 27 00:15:59.147265085 (512h46m59.01834813s) [seg_pos={0 0}, upload=449972]: Done begin upload segment. Dec 27 00:15:59.147295914 (512h46m59.018378959s) [noise=true , address=145.79.227.46:4601 , piece_id=FW4SHC75P33CDXR2TYFWGIHOBFINYWL4B5B5WX5URMNFQYMSZH4A, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.147319159 (512h46m59.018402204s) [noise=true , address=45.91.116.6:4975 , piece_id=EBZCOPRTTJG4WOQALJHIW7EZK5SZBHJDKSSXDDTZHD23AHYOHPCQ, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.147888392 (512h46m59.018971437s) [noise=true , address=45.91.116.9:4264 , piece_id=CUK3YC72WCBNK5S2C4A4CRQWA7ISDMFXONVW3E4HXIDJCPT2NP6A, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.148346334 (512h46m59.019429379s) [noise=true , address=145.79.227.5:4222 , piece_id=AEH2PTPLHIZUEJV635KQ6GBJA33REYU4EUHDYK75BNSUBCWWM5RQ, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.148817141 (512h46m59.019900186s) [noise=true , address=145.79.227.47:4265 , piece_id=MAKLCUQ7GV2SGLLJE73EEFSFPUD4HXDFPQKVUIKKTMHY25FJTE3Q, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.149293538 (512h46m59.020376573s) [noise=true , address=145.79.227.3:4151 , piece_id=B7JH6YDCAOQNIFG42IGEAW3JWVX4DRB2PO2UNA5R7YNBYUKVKTTQ, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.149931824 (512h46m59.021014859s) [noise=true , address=145.79.227.36:4518 , piece_id=DE3O4GUAHMQ25WIBUT4PVH2L427KD2QC4362ZJHJ4S2AOPSLDMTA, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.150533028 (512h46m59.021616073s) [noise=true , address=145.79.227.50:4375 , piece_id=J6TMBQDMVCA7SRHMIH526LVPCCTDOADECOI3QJDKA75UTSOGLQ4A, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.150550661 (512h46m59.021633716s) [noise=true , address=145.79.227.14:4837 , piece_id=2UGRZP6VJMORNORY6NXXECZKCE3XUUX4IT6GS2TIFBPVJ7SOE5TQ, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.151068207 (512h46m59.022151242s) [noise=true , address=77.90.31.15:4700 , piece_id=CV5PAC7KCIHUJSHH6R3W3YQ53RMK4YUIF2HWYMPHP2PK5P67Q6AA, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.151599478 (512h46m59.022682513s) [noise=true , address=145.79.227.42:4123 , piece_id=K3GYGSH6WAAOYFUNGKG3ZJLBUTHAEWWX3JHXPQKZDAOW2FD44Q4Q, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.152230720 (512h46m59.023313796s) [noise=true , address=145.79.227.37:4567 , piece_id=TBPFIBTMLDFKTJ4LNKSUBZ6MTR3WH4BWMMQM77U3EOVO4SZW5I5A, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.153146635 (512h46m59.02422967s) [noise=true , address=45.91.116.5:4297 , piece_id=PLVVBC3DBKNXMO6MSWPWAGAIWHGXJFVTY2K2EB6MV5MYXXFVNC4Q, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.154944888 (512h46m59.026027944s) [noise=true , address=45.91.116.11:4119 , piece_id=PDFKHLRFJTTAXUU4R7FL3VRAAH7QWRBGSJIHAT7KKHBOORMDCHHA, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.156795182 (512h46m59.027878227s) [noise=true , address=145.79.227.17:4971 , piece_id=NAFDG2U3MMWFMY7USYNVNYU5VP3QFT3YVADEJ2O7VHUCKSF3IRTA, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:15:59.169324770 (512h46m59.040407875s) [noise=true , address=145.79.227.48:4302 , piece_id=YDJ63ZQF45A2JCCKEMTXCR3IJWK3WX7C2GM2EMLI7JP7LO4MAOFA, seg_pos={0 0}, upload=449972]: Uploading piece... Dec 27 00:16:00.761283060 (512h47m0.632366105s) [upload=449972]: Next returned nil segment Dec 27 00:16:00.761298759 (512h47m0.632381804s) [upload=449972]: Waiting for error group managing segments... Dec 27 00:16:01.147955290 (512h47m1.019038335s) [noise=true , address=145.79.227.14:4837 , piece_id=2UGRZP6VJMORNORY6NXXECZKCE3XUUX4IT6GS2TIFBPVJ7SOE5TQ, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.148866495 (512h47m1.01994953s) [noise=true , address=145.79.227.47:4265 , piece_id=MAKLCUQ7GV2SGLLJE73EEFSFPUD4HXDFPQKVUIKKTMHY25FJTE3Q, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.155582503 (512h47m1.026665538s) [noise=true , address=145.79.227.5:4222 , piece_id=AEH2PTPLHIZUEJV635KQ6GBJA33REYU4EUHDYK75BNSUBCWWM5RQ, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.159391706 (512h47m1.030474751s) [noise=true , address=145.79.227.50:4375 , piece_id=J6TMBQDMVCA7SRHMIH526LVPCCTDOADECOI3QJDKA75UTSOGLQ4A, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.159434457 (512h47m1.030517502s) [noise=true , address=145.79.227.46:4601 , piece_id=FW4SHC75P33CDXR2TYFWGIHOBFINYWL4B5B5WX5URMNFQYMSZH4A, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.169768094 (512h47m1.040851129s) [noise=true , address=145.79.227.48:4302 , piece_id=YDJ63ZQF45A2JCCKEMTXCR3IJWK3WX7C2GM2EMLI7JP7LO4MAOFA, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.182193884 (512h47m1.053276929s) [noise=true , address=145.79.227.42:4123 , piece_id=K3GYGSH6WAAOYFUNGKG3ZJLBUTHAEWWX3JHXPQKZDAOW2FD44Q4Q, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.187378066 (512h47m1.058461111s) [noise=true , address=145.79.227.36:4518 , piece_id=DE3O4GUAHMQ25WIBUT4PVH2L427KD2QC4362ZJHJ4S2AOPSLDMTA, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.190972200 (512h47m1.062055235s) [noise=true , address=145.79.227.3:4151 , piece_id=B7JH6YDCAOQNIFG42IGEAW3JWVX4DRB2PO2UNA5R7YNBYUKVKTTQ, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.192681855 (512h47m1.06376489s) [noise=true , address=145.79.227.37:4567 , piece_id=TBPFIBTMLDFKTJ4LNKSUBZ6MTR3WH4BWMMQM77U3EOVO4SZW5I5A, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.195930270 (512h47m1.067013315s) [noise=true , address=145.79.227.17:4971 , piece_id=NAFDG2U3MMWFMY7USYNVNYU5VP3QFT3YVADEJ2O7VHUCKSF3IRTA, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.394477788 (512h47m1.265560833s) [noise=true , address=45.91.116.5:4297 , piece_id=PLVVBC3DBKNXMO6MSWPWAGAIWHGXJFVTY2K2EB6MV5MYXXFVNC4Q, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.430641829 (512h47m1.301724874s) [noise=true , address=45.91.116.6:4975 , piece_id=EBZCOPRTTJG4WOQALJHIW7EZK5SZBHJDKSSXDDTZHD23AHYOHPCQ, seg_pos={0 0}, upload=449972]: Done uploading piece. err: <nil> Dec 27 00:16:01.430712834 (512h47m1.301795889s) [seg_pos={0 0}, upload=449972]: Segment reached optimal threshold of 13 pieces. Dec 27 00:16:01.430955266 (512h47m1.302038301s) [noise=true , address=45.91.116.9:4264 , piece_id=CUK3YC72WCBNK5S2C4A4CRQWA7ISDMFXONVW3E4HXIDJCPT2NP6A, seg_pos={0 0}, upload=449972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 27 00:16:01.430967218 (512h47m1.302050253s) [noise=true , address=45.91.116.9:4264 , piece_id=CUK3YC72WCBNK5S2C4A4CRQWA7ISDMFXONVW3E4HXIDJCPT2NP6A, seg_pos={0 0}, upload=449972]: Long tail cancellation due to optimal threshold Dec 27 00:16:01.431218727 (512h47m1.302301762s) [noise=true , address=77.90.31.15:4700 , piece_id=CV5PAC7KCIHUJSHH6R3W3YQ53RMK4YUIF2HWYMPHP2PK5P67Q6AA, seg_pos={0 0}, upload=449972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 00:16:01.431225751 (512h47m1.302308786s) [noise=true , address=77.90.31.15:4700 , piece_id=CV5PAC7KCIHUJSHH6R3W3YQ53RMK4YUIF2HWYMPHP2PK5P67Q6AA, seg_pos={0 0}, upload=449972]: Long tail cancellation due to optimal threshold Dec 27 00:16:01.431463153 (512h47m1.302546189s) [noise=true , address=45.91.116.11:4119 , piece_id=PDFKHLRFJTTAXUU4R7FL3VRAAH7QWRBGSJIHAT7KKHBOORMDCHHA, seg_pos={0 0}, upload=449972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 27 00:16:01.431484123 (512h47m1.302567188s) [noise=true , address=45.91.116.11:4119 , piece_id=PDFKHLRFJTTAXUU4R7FL3VRAAH7QWRBGSJIHAT7KKHBOORMDCHHA, seg_pos={0 0}, upload=449972]: Long tail cancellation due to optimal threshold Dec 27 00:16:01.431520432 (512h47m1.302603477s) [seg_pos={0 0}, upload=449972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:16:01.431647144 (512h47m1.302730189s) [upload=449972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:16:01.431656942 (512h47m1.302739987s) [upload=449972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:16:01.713942788 (512h47m1.585025833s) [upload=449972]: Done uploading segments... Dec 27 00:16:01.713968075 (512h47m1.58505112s) [upload=449972]: Upload finished. err: <nil> Duration: 30.002156951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X..XX.XX..XXXX..X.XX...X.........X.........X.......X.XX.X] duration: 30.002860773s trace: https://jaeger.dev.storj.io/trace/61e4ac4e32e90a60 FAIL (2025-12-27 00:15:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: DUQARSVFVIXRYZ3FF77JAU3OQM7AUOR2DI2OEMGWK3UEDZMQXHDQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: EGSIZHAHGUPWPIO5GOXQ7OEFUNP4QRRPCGL4ZIPH2INJV6SMIJCQ) context deadline exceeded ======LOGS====== Dec 27 00:15:28.693279598 (512h46m28.564362633s) [upload=449966]: Starting upload Dec 27 00:15:28.693374730 (512h46m28.564457775s) [upload=449966]: Done starting upload Dec 27 00:15:28.693489247 (512h46m28.564572292s) [upload=449966]: Uploading segments... Dec 27 00:15:28.693549462 (512h46m28.564632507s) [upload=449966]: Got next segment. Inline: false Dec 27 00:15:28.693589478 (512h46m28.564672523s) [upload=449966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:15:28.693597153 (512h46m28.564680188s) [upload=449966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:15:28.939187136 (512h46m28.810270181s) [seg_pos={0 0}, upload=449966]: Begin upload segment... Dec 27 00:15:28.939254745 (512h46m28.81033779s) [seg_pos={0 0}, upload=449966]: Done begin upload segment. Dec 27 00:15:28.939292557 (512h46m28.810375602s) [noise=true , address=145.79.227.41:4085 , piece_id=FCWEVLVQZ34WH7EZETCGI5XFZBKKOONH6524VIYCCXWJ4SURQZCA, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.939364965 (512h46m28.81044803s) [noise=true , address=145.79.227.13:4802 , piece_id=W3NTN3PLFIYEX3GLMF65G65LQRX2H7QXQTMHLUIPKRAX3BE5EI3A, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.940588645 (512h46m28.81167169s) [noise=true , address=145.79.227.3:4149 , piece_id=FQPGP5CISW5PQTXQNTQZSDY4E6CXVT3TWFIAD6DWCM3NU6Z6ILJQ, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.941315539 (512h46m28.812398584s) [noise=true , address=145.79.227.35:4482 , piece_id=X44VZEJB2KBTJO7LOBO6FIWB4M5KDROGDXLJ4KOXKAVXD675YSMA, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.942048495 (512h46m28.81313153s) [noise=true , address=45.91.116.13:4155 , piece_id=4OWWZJ5XWQ4YGDQX4OSDJ3EYZR3HKP6IXW2ZN7RLRWLHCRTBYQKQ, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.942164676 (512h46m28.813247721s) [noise=true , address=145.79.227.50:4374 , piece_id=NR3NOMSFQKDNHIY4UOO3PHF77A5CWWKKZZWBGMJ7V4GNEMXG4ECA, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.942745181 (512h46m28.813828226s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUWVN7PJ7CFFYDC2P3MQWEOCK7IYUD4COYERLUWFZHYBOEABESHQ, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.942902410 (512h46m28.813985445s) [noise=true , address=77.90.31.13:4627 , piece_id=S6GXLZBUNVY7XKL52TF2HQJUV2ICXF7NPSNYZP6OADPK65RNODDA, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.943271063 (512h46m28.814354108s) [noise=true , address=77.90.31.21:4880 , piece_id=DUQARSVFVIXRYZ3FF77JAU3OQM7AUOR2DI2OEMGWK3UEDZMQXHDQ, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.943629765 (512h46m28.81471281s) [noise=true , address=77.90.31.10:4520 , piece_id=EGSIZHAHGUPWPIO5GOXQ7OEFUNP4QRRPCGL4ZIPH2INJV6SMIJCQ, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.943808105 (512h46m28.81489114s) [noise=true , address=145.79.227.49:4339 , piece_id=3RMBVLAA3BN5YLIBOK26PMXHM2OQPKPRZMIR5B3HPOFC7AAQZ4IA, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.944186295 (512h46m28.81526934s) [noise=true , address=145.79.227.52:4446 , piece_id=MRV7XSIS4BTZHSP7FXI3US3EFAMY4BG4W7ILUBKE76S67ATAVBJQ, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.944328826 (512h46m28.815411861s) [noise=true , address=145.79.227.42:4121 , piece_id=MFUXZKGX4ZPAOTQJZEQ7EXOV57NUEBLFHISAV6SJ5M5S3EZM6YPQ, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.944862764 (512h46m28.815945809s) [noise=true , address=145.79.227.15:4875 , piece_id=Q5VPE6SKO55ULE7DZS76YZ5AOJ7OOOHE3NN2JH6K6QFBVTBQTO4A, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.945373255 (512h46m28.8164563s) [noise=true , address=145.79.227.23:4545 , piece_id=FHHA4WEYTWVNKJKF7OAZUXLLWIWKDH3IZATMBYGNHCFEPBGJSVGQ, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:28.945961565 (512h46m28.817044611s) [noise=true , address=77.90.31.19:4844 , piece_id=RVGONJ626OQTYNLFIH5H4TLFTABEYVWQPST3EOYYK7KJA3M7FCYA, seg_pos={0 0}, upload=449966]: Uploading piece... Dec 27 00:15:30.562486031 (512h46m30.433569066s) [upload=449966]: Next returned nil segment Dec 27 00:15:30.562504987 (512h46m30.433588022s) [upload=449966]: Waiting for error group managing segments... Dec 27 00:15:30.946415498 (512h46m30.817498543s) [noise=true , address=145.79.227.23:4545 , piece_id=FHHA4WEYTWVNKJKF7OAZUXLLWIWKDH3IZATMBYGNHCFEPBGJSVGQ, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:30.946946750 (512h46m30.818029785s) [noise=true , address=145.79.227.13:4802 , piece_id=W3NTN3PLFIYEX3GLMF65G65LQRX2H7QXQTMHLUIPKRAX3BE5EI3A, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:30.947455017 (512h46m30.818538072s) [noise=true , address=145.79.227.50:4374 , piece_id=NR3NOMSFQKDNHIY4UOO3PHF77A5CWWKKZZWBGMJ7V4GNEMXG4ECA, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:30.947800395 (512h46m30.81888344s) [noise=true , address=145.79.227.15:4875 , piece_id=Q5VPE6SKO55ULE7DZS76YZ5AOJ7OOOHE3NN2JH6K6QFBVTBQTO4A, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:30.951962671 (512h46m30.823045716s) [noise=true , address=145.79.227.3:4149 , piece_id=FQPGP5CISW5PQTXQNTQZSDY4E6CXVT3TWFIAD6DWCM3NU6Z6ILJQ, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:30.952007326 (512h46m30.823090361s) [noise=true , address=145.79.227.41:4085 , piece_id=FCWEVLVQZ34WH7EZETCGI5XFZBKKOONH6524VIYCCXWJ4SURQZCA, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:30.955984930 (512h46m30.827067975s) [noise=true , address=145.79.227.42:4121 , piece_id=MFUXZKGX4ZPAOTQJZEQ7EXOV57NUEBLFHISAV6SJ5M5S3EZM6YPQ, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:30.959181196 (512h46m30.830264231s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUWVN7PJ7CFFYDC2P3MQWEOCK7IYUD4COYERLUWFZHYBOEABESHQ, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:30.972914054 (512h46m30.843997089s) [noise=true , address=145.79.227.52:4446 , piece_id=MRV7XSIS4BTZHSP7FXI3US3EFAMY4BG4W7ILUBKE76S67ATAVBJQ, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:30.988750278 (512h46m30.859833313s) [noise=true , address=145.79.227.49:4339 , piece_id=3RMBVLAA3BN5YLIBOK26PMXHM2OQPKPRZMIR5B3HPOFC7AAQZ4IA, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:31.124685014 (512h46m30.995768089s) [noise=true , address=77.90.31.21:4880 , piece_id=DUQARSVFVIXRYZ3FF77JAU3OQM7AUOR2DI2OEMGWK3UEDZMQXHDQ, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:31.189391863 (512h46m31.060474908s) [noise=true , address=77.90.31.10:4520 , piece_id=EGSIZHAHGUPWPIO5GOXQ7OEFUNP4QRRPCGL4ZIPH2INJV6SMIJCQ, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:31.956734675 (512h46m31.82781771s) [noise=true , address=45.91.116.13:4155 , piece_id=4OWWZJ5XWQ4YGDQX4OSDJ3EYZR3HKP6IXW2ZN7RLRWLHCRTBYQKQ, seg_pos={0 0}, upload=449966]: Done uploading piece. err: <nil> Dec 27 00:15:31.956756086 (512h46m31.827839121s) [seg_pos={0 0}, upload=449966]: Segment reached optimal threshold of 13 pieces. Dec 27 00:15:31.956973079 (512h46m31.828056124s) [noise=true , address=145.79.227.35:4482 , piece_id=X44VZEJB2KBTJO7LOBO6FIWB4M5KDROGDXLJ4KOXKAVXD675YSMA, seg_pos={0 0}, upload=449966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 27 00:15:31.956984070 (512h46m31.828067116s) [noise=true , address=145.79.227.35:4482 , piece_id=X44VZEJB2KBTJO7LOBO6FIWB4M5KDROGDXLJ4KOXKAVXD675YSMA, seg_pos={0 0}, upload=449966]: Long tail cancellation due to optimal threshold Dec 27 00:15:31.956993498 (512h46m31.828076543s) [noise=true , address=77.90.31.13:4627 , piece_id=S6GXLZBUNVY7XKL52TF2HQJUV2ICXF7NPSNYZP6OADPK65RNODDA, seg_pos={0 0}, upload=449966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58568->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58568->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->77.90.31.13:4627: use of closed network connection Dec 27 00:15:31.957002164 (512h46m31.828085209s) [noise=true , address=77.90.31.13:4627 , piece_id=S6GXLZBUNVY7XKL52TF2HQJUV2ICXF7NPSNYZP6OADPK65RNODDA, seg_pos={0 0}, upload=449966]: Long tail cancellation due to optimal threshold Dec 27 00:15:31.957218827 (512h46m31.828301862s) [noise=true , address=77.90.31.19:4844 , piece_id=RVGONJ626OQTYNLFIH5H4TLFTABEYVWQPST3EOYYK7KJA3M7FCYA, seg_pos={0 0}, upload=449966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 27 00:15:31.957229428 (512h46m31.828312473s) [noise=true , address=77.90.31.19:4844 , piece_id=RVGONJ626OQTYNLFIH5H4TLFTABEYVWQPST3EOYYK7KJA3M7FCYA, seg_pos={0 0}, upload=449966]: Long tail cancellation due to optimal threshold Dec 27 00:15:31.957250868 (512h46m31.828333903s) [seg_pos={0 0}, upload=449966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:15:31.957332704 (512h46m31.828415739s) [upload=449966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:15:31.957339297 (512h46m31.828422342s) [upload=449966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:15:32.256937657 (512h46m32.128020692s) [upload=449966]: Done uploading segments... Dec 27 00:15:32.256965190 (512h46m32.128048225s) [upload=449966]: Upload finished. err: <nil> Duration: 30.00288562s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 261.428311ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:15:38 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 261.46987ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X..XX.XX..XXXX..X.XX...X.........X.........X.......X.XX.] duration: 6.283915213s trace: https://jaeger.dev.storj.io/trace/28578c55c6916459 FAIL (2025-12-27 00:15:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 5TSIKCVGTKU2XFPZSRB6NWYEUCCZKMDU4ECMWSEEVZKNNSQIJR7A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: H43Y237LYXCETGDM3C5DNX66YZIMILJAD6T5F6VPNQXKINX5JI4Q) context deadline exceeded ======LOGS====== Dec 27 00:13:47.423944886 (512h44m47.295027931s) [upload=449937]: Starting upload Dec 27 00:13:47.424003267 (512h44m47.295086312s) [upload=449937]: Done starting upload Dec 27 00:13:47.424070113 (512h44m47.295153149s) [upload=449937]: Uploading segments... Dec 27 00:13:47.424179422 (512h44m47.295262468s) [upload=449937]: Got next segment. Inline: false Dec 27 00:13:47.424226382 (512h44m47.295309417s) [upload=449937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:13:47.424243514 (512h44m47.295326549s) [upload=449937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:13:47.671388768 (512h44m47.542471813s) [seg_pos={0 0}, upload=449937]: Begin upload segment... Dec 27 00:13:47.671444174 (512h44m47.542527209s) [seg_pos={0 0}, upload=449937]: Done begin upload segment. Dec 27 00:13:47.671505009 (512h44m47.542588054s) [noise=true , address=145.79.227.15:4875 , piece_id=G7MFJ3GU3I5YLGQQNSTTS6ZYMPZC6C4AGRW2TPF5GZFSPX2NO4ZQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.671536760 (512h44m47.542619795s) [noise=true , address=45.91.116.11:4119 , piece_id=KHDUBLEBIX74USWMPWOSDS5I2S2L5XFOUH44V7T6C73DT7MAN3QA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.671605891 (512h44m47.542688957s) [noise=true , address=77.90.31.15:4700 , piece_id=5TSIKCVGTKU2XFPZSRB6NWYEUCCZKMDU4ECMWSEEVZKNNSQIJR7A, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.672214541 (512h44m47.543297586s) [noise=true , address=45.91.116.6:4975 , piece_id=P4HUXGXKJKH2NCU5TRECTIBM736FDQWJDH35ZO7SYBZSJROGEE2Q, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.672766933 (512h44m47.543849988s) [noise=true , address=145.79.227.48:4301 , piece_id=RHVWOXJKTZV37XFPZHU6QJMH7M6DJ6JYPIENKI6XR3CVCTPO5TRQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.673151675 (512h44m47.544234721s) [noise=true , address=145.79.227.39:4015 , piece_id=PXTJTMTS4SI7DV4HVIJ5ZBDOG4ZAT5XWD3WVS6KBXETY53DBYYMA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.673545695 (512h44m47.544628731s) [noise=true , address=45.91.116.13:4154 , piece_id=PEAMQR7KDE2DB6YVI7NY657PMQK6QFTDCBTYHUIQSF2VQHK53OMA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.673960545 (512h44m47.545043581s) [noise=true , address=45.91.116.7:4010 , piece_id=4SHFHALVU3MOIN2I5L4F2AW4GUTCN42HZ7WURLQ7HSLS5K5CNOKA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.674462711 (512h44m47.545545756s) [noise=true , address=45.91.116.14:4189 , piece_id=JR7L3AAG7TQZSDGZOD7HH76YBS5BYM7LIRF2R37VPIH3DC4T5IEA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.674915554 (512h44m47.545998599s) [noise=true , address=145.79.227.14:4837 , piece_id=IRPXJ4UBRV2H2OYJBUI4TDMAEB6ZMD6IC7QQ3C6OWAKQJBC2RK7Q, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.676560445 (512h44m47.547643501s) [noise=true , address=45.91.116.5:4297 , piece_id=6WYTREOPM7KK4XYKB2D3GCIFOYDMGLSLQ3CDJLIT3RTSNSCTPDNA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.678203133 (512h44m47.549286178s) [noise=true , address=145.79.227.49:4338 , piece_id=3YKBYIUJOPG7LMLV7PZOAE3UKEIPMHEPTWS3VFNXXG2HQSIU2DOQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.680057585 (512h44m47.55114062s) [noise=true , address=77.90.31.12:4589 , piece_id=H43Y237LYXCETGDM3C5DNX66YZIMILJAD6T5F6VPNQXKINX5JI4Q, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.681557281 (512h44m47.552640316s) [noise=true , address=145.79.227.37:4567 , piece_id=GIPWQXA5ZNNZOIEZ5Y2HSXKQVDNGQGVM3FZFRG325YV73T7UAPZQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.683046325 (512h44m47.55412937s) [noise=true , address=145.79.227.3:4150 , piece_id=7KHRCC4JQS2UQB5BEVNJTQSLPR6M62F4EZKPW6SAGNHQITAGNJCQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.684568444 (512h44m47.555651479s) [noise=true , address=45.91.116.10:4084 , piece_id=HNUW4H7NRS4RME2BEZBSO3XCGPG4DFUDNPRBRQ5AV7IRLNOF24LQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:49.550664735 (512h44m49.42174778s) [upload=449937]: Next returned nil segment Dec 27 00:13:49.550680444 (512h44m49.421763479s) [upload=449937]: Waiting for error group managing segments... Dec 27 00:13:49.863591495 (512h44m49.73467453s) [noise=true , address=145.79.227.48:4301 , piece_id=RHVWOXJKTZV37XFPZHU6QJMH7M6DJ6JYPIENKI6XR3CVCTPO5TRQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.872909648 (512h44m49.743992693s) [noise=true , address=145.79.227.39:4015 , piece_id=PXTJTMTS4SI7DV4HVIJ5ZBDOG4ZAT5XWD3WVS6KBXETY53DBYYMA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.881774617 (512h44m49.752857663s) [noise=true , address=145.79.227.3:4150 , piece_id=7KHRCC4JQS2UQB5BEVNJTQSLPR6M62F4EZKPW6SAGNHQITAGNJCQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.887789911 (512h44m49.758872956s) [noise=true , address=45.91.116.7:4010 , piece_id=4SHFHALVU3MOIN2I5L4F2AW4GUTCN42HZ7WURLQ7HSLS5K5CNOKA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.907733005 (512h44m49.77881605s) [noise=true , address=145.79.227.14:4837 , piece_id=IRPXJ4UBRV2H2OYJBUI4TDMAEB6ZMD6IC7QQ3C6OWAKQJBC2RK7Q, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.951273191 (512h44m49.822356236s) [noise=true , address=77.90.31.15:4700 , piece_id=5TSIKCVGTKU2XFPZSRB6NWYEUCCZKMDU4ECMWSEEVZKNNSQIJR7A, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.958632104 (512h44m49.829715149s) [noise=true , address=145.79.227.37:4567 , piece_id=GIPWQXA5ZNNZOIEZ5Y2HSXKQVDNGQGVM3FZFRG325YV73T7UAPZQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.963351019 (512h44m49.834434054s) [noise=true , address=45.91.116.14:4189 , piece_id=JR7L3AAG7TQZSDGZOD7HH76YBS5BYM7LIRF2R37VPIH3DC4T5IEA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.003318381 (512h44m49.874401416s) [noise=true , address=45.91.116.13:4154 , piece_id=PEAMQR7KDE2DB6YVI7NY657PMQK6QFTDCBTYHUIQSF2VQHK53OMA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.010253136 (512h44m49.881336171s) [noise=true , address=45.91.116.5:4297 , piece_id=6WYTREOPM7KK4XYKB2D3GCIFOYDMGLSLQ3CDJLIT3RTSNSCTPDNA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.262964799 (512h44m50.134047834s) [noise=true , address=45.91.116.6:4975 , piece_id=P4HUXGXKJKH2NCU5TRECTIBM736FDQWJDH35ZO7SYBZSJROGEE2Q, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.518456817 (512h44m50.389539862s) [noise=true , address=145.79.227.49:4338 , piece_id=3YKBYIUJOPG7LMLV7PZOAE3UKEIPMHEPTWS3VFNXXG2HQSIU2DOQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.743293642 (512h44m50.614376677s) [noise=true , address=77.90.31.12:4589 , piece_id=H43Y237LYXCETGDM3C5DNX66YZIMILJAD6T5F6VPNQXKINX5JI4Q, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.743313079 (512h44m50.614396114s) [seg_pos={0 0}, upload=449937]: Segment reached optimal threshold of 13 pieces. Dec 27 00:13:50.743502049 (512h44m50.614585114s) [noise=true , address=145.79.227.15:4875 , piece_id=G7MFJ3GU3I5YLGQQNSTTS6ZYMPZC6C4AGRW2TPF5GZFSPX2NO4ZQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 27 00:13:50.743512660 (512h44m50.614595705s) [noise=true , address=145.79.227.15:4875 , piece_id=G7MFJ3GU3I5YLGQQNSTTS6ZYMPZC6C4AGRW2TPF5GZFSPX2NO4ZQ, seg_pos={0 0}, upload=449937]: Long tail cancellation due to optimal threshold Dec 27 00:13:50.743680720 (512h44m50.614763765s) [noise=true , address=45.91.116.11:4119 , piece_id=KHDUBLEBIX74USWMPWOSDS5I2S2L5XFOUH44V7T6C73DT7MAN3QA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 27 00:13:50.743687723 (512h44m50.614770758s) [noise=true , address=45.91.116.11:4119 , piece_id=KHDUBLEBIX74USWMPWOSDS5I2S2L5XFOUH44V7T6C73DT7MAN3QA, seg_pos={0 0}, upload=449937]: Long tail cancellation due to optimal threshold Dec 27 00:13:50.743715847 (512h44m50.614798892s) [noise=true , address=45.91.116.10:4084 , piece_id=HNUW4H7NRS4RME2BEZBSO3XCGPG4DFUDNPRBRQ5AV7IRLNOF24LQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: read tcp 188.166.225.170:39470->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 00:13:50.743734261 (512h44m50.614817306s) [noise=true , address=45.91.116.10:4084 , piece_id=HNUW4H7NRS4RME2BEZBSO3XCGPG4DFUDNPRBRQ5AV7IRLNOF24LQ, seg_pos={0 0}, upload=449937]: Long tail cancellation due to optimal threshold Dec 27 00:13:50.743761203 (512h44m50.614844238s) [seg_pos={0 0}, upload=449937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:13:50.743932890 (512h44m50.615015925s) [upload=449937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:13:50.743939793 (512h44m50.615022838s) [upload=449937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:13:51.035407432 (512h44m50.906490468s) [upload=449937]: Done uploading segments... Dec 27 00:13:51.035427089 (512h44m50.906510124s) [upload=449937]: Upload finished. err: <nil> Duration: 6.283942274s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 254.273688ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:14:34 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 254.300769ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....X..XX.XX..XXXX..X.XX...X.........X.........X.......X.XX] duration: 30.003692592s trace: https://jaeger.dev.storj.io/trace/28578c55c6916459 FAIL (2025-12-27 00:14:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 5TSIKCVGTKU2XFPZSRB6NWYEUCCZKMDU4ECMWSEEVZKNNSQIJR7A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: H43Y237LYXCETGDM3C5DNX66YZIMILJAD6T5F6VPNQXKINX5JI4Q) context deadline exceeded ======LOGS====== Dec 27 00:13:47.423944886 (512h44m47.295027931s) [upload=449937]: Starting upload Dec 27 00:13:47.424003267 (512h44m47.295086312s) [upload=449937]: Done starting upload Dec 27 00:13:47.424070113 (512h44m47.295153149s) [upload=449937]: Uploading segments... Dec 27 00:13:47.424179422 (512h44m47.295262468s) [upload=449937]: Got next segment. Inline: false Dec 27 00:13:47.424226382 (512h44m47.295309417s) [upload=449937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:13:47.424243514 (512h44m47.295326549s) [upload=449937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:13:47.671388768 (512h44m47.542471813s) [seg_pos={0 0}, upload=449937]: Begin upload segment... Dec 27 00:13:47.671444174 (512h44m47.542527209s) [seg_pos={0 0}, upload=449937]: Done begin upload segment. Dec 27 00:13:47.671505009 (512h44m47.542588054s) [noise=true , address=145.79.227.15:4875 , piece_id=G7MFJ3GU3I5YLGQQNSTTS6ZYMPZC6C4AGRW2TPF5GZFSPX2NO4ZQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.671536760 (512h44m47.542619795s) [noise=true , address=45.91.116.11:4119 , piece_id=KHDUBLEBIX74USWMPWOSDS5I2S2L5XFOUH44V7T6C73DT7MAN3QA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.671605891 (512h44m47.542688957s) [noise=true , address=77.90.31.15:4700 , piece_id=5TSIKCVGTKU2XFPZSRB6NWYEUCCZKMDU4ECMWSEEVZKNNSQIJR7A, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.672214541 (512h44m47.543297586s) [noise=true , address=45.91.116.6:4975 , piece_id=P4HUXGXKJKH2NCU5TRECTIBM736FDQWJDH35ZO7SYBZSJROGEE2Q, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.672766933 (512h44m47.543849988s) [noise=true , address=145.79.227.48:4301 , piece_id=RHVWOXJKTZV37XFPZHU6QJMH7M6DJ6JYPIENKI6XR3CVCTPO5TRQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.673151675 (512h44m47.544234721s) [noise=true , address=145.79.227.39:4015 , piece_id=PXTJTMTS4SI7DV4HVIJ5ZBDOG4ZAT5XWD3WVS6KBXETY53DBYYMA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.673545695 (512h44m47.544628731s) [noise=true , address=45.91.116.13:4154 , piece_id=PEAMQR7KDE2DB6YVI7NY657PMQK6QFTDCBTYHUIQSF2VQHK53OMA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.673960545 (512h44m47.545043581s) [noise=true , address=45.91.116.7:4010 , piece_id=4SHFHALVU3MOIN2I5L4F2AW4GUTCN42HZ7WURLQ7HSLS5K5CNOKA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.674462711 (512h44m47.545545756s) [noise=true , address=45.91.116.14:4189 , piece_id=JR7L3AAG7TQZSDGZOD7HH76YBS5BYM7LIRF2R37VPIH3DC4T5IEA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.674915554 (512h44m47.545998599s) [noise=true , address=145.79.227.14:4837 , piece_id=IRPXJ4UBRV2H2OYJBUI4TDMAEB6ZMD6IC7QQ3C6OWAKQJBC2RK7Q, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.676560445 (512h44m47.547643501s) [noise=true , address=45.91.116.5:4297 , piece_id=6WYTREOPM7KK4XYKB2D3GCIFOYDMGLSLQ3CDJLIT3RTSNSCTPDNA, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.678203133 (512h44m47.549286178s) [noise=true , address=145.79.227.49:4338 , piece_id=3YKBYIUJOPG7LMLV7PZOAE3UKEIPMHEPTWS3VFNXXG2HQSIU2DOQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.680057585 (512h44m47.55114062s) [noise=true , address=77.90.31.12:4589 , piece_id=H43Y237LYXCETGDM3C5DNX66YZIMILJAD6T5F6VPNQXKINX5JI4Q, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.681557281 (512h44m47.552640316s) [noise=true , address=145.79.227.37:4567 , piece_id=GIPWQXA5ZNNZOIEZ5Y2HSXKQVDNGQGVM3FZFRG325YV73T7UAPZQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.683046325 (512h44m47.55412937s) [noise=true , address=145.79.227.3:4150 , piece_id=7KHRCC4JQS2UQB5BEVNJTQSLPR6M62F4EZKPW6SAGNHQITAGNJCQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:47.684568444 (512h44m47.555651479s) [noise=true , address=45.91.116.10:4084 , piece_id=HNUW4H7NRS4RME2BEZBSO3XCGPG4DFUDNPRBRQ5AV7IRLNOF24LQ, seg_pos={0 0}, upload=449937]: Uploading piece... Dec 27 00:13:49.550664735 (512h44m49.42174778s) [upload=449937]: Next returned nil segment Dec 27 00:13:49.550680444 (512h44m49.421763479s) [upload=449937]: Waiting for error group managing segments... Dec 27 00:13:49.863591495 (512h44m49.73467453s) [noise=true , address=145.79.227.48:4301 , piece_id=RHVWOXJKTZV37XFPZHU6QJMH7M6DJ6JYPIENKI6XR3CVCTPO5TRQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.872909648 (512h44m49.743992693s) [noise=true , address=145.79.227.39:4015 , piece_id=PXTJTMTS4SI7DV4HVIJ5ZBDOG4ZAT5XWD3WVS6KBXETY53DBYYMA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.881774617 (512h44m49.752857663s) [noise=true , address=145.79.227.3:4150 , piece_id=7KHRCC4JQS2UQB5BEVNJTQSLPR6M62F4EZKPW6SAGNHQITAGNJCQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.887789911 (512h44m49.758872956s) [noise=true , address=45.91.116.7:4010 , piece_id=4SHFHALVU3MOIN2I5L4F2AW4GUTCN42HZ7WURLQ7HSLS5K5CNOKA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.907733005 (512h44m49.77881605s) [noise=true , address=145.79.227.14:4837 , piece_id=IRPXJ4UBRV2H2OYJBUI4TDMAEB6ZMD6IC7QQ3C6OWAKQJBC2RK7Q, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.951273191 (512h44m49.822356236s) [noise=true , address=77.90.31.15:4700 , piece_id=5TSIKCVGTKU2XFPZSRB6NWYEUCCZKMDU4ECMWSEEVZKNNSQIJR7A, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.958632104 (512h44m49.829715149s) [noise=true , address=145.79.227.37:4567 , piece_id=GIPWQXA5ZNNZOIEZ5Y2HSXKQVDNGQGVM3FZFRG325YV73T7UAPZQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:49.963351019 (512h44m49.834434054s) [noise=true , address=45.91.116.14:4189 , piece_id=JR7L3AAG7TQZSDGZOD7HH76YBS5BYM7LIRF2R37VPIH3DC4T5IEA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.003318381 (512h44m49.874401416s) [noise=true , address=45.91.116.13:4154 , piece_id=PEAMQR7KDE2DB6YVI7NY657PMQK6QFTDCBTYHUIQSF2VQHK53OMA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.010253136 (512h44m49.881336171s) [noise=true , address=45.91.116.5:4297 , piece_id=6WYTREOPM7KK4XYKB2D3GCIFOYDMGLSLQ3CDJLIT3RTSNSCTPDNA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.262964799 (512h44m50.134047834s) [noise=true , address=45.91.116.6:4975 , piece_id=P4HUXGXKJKH2NCU5TRECTIBM736FDQWJDH35ZO7SYBZSJROGEE2Q, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.518456817 (512h44m50.389539862s) [noise=true , address=145.79.227.49:4338 , piece_id=3YKBYIUJOPG7LMLV7PZOAE3UKEIPMHEPTWS3VFNXXG2HQSIU2DOQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.743293642 (512h44m50.614376677s) [noise=true , address=77.90.31.12:4589 , piece_id=H43Y237LYXCETGDM3C5DNX66YZIMILJAD6T5F6VPNQXKINX5JI4Q, seg_pos={0 0}, upload=449937]: Done uploading piece. err: <nil> Dec 27 00:13:50.743313079 (512h44m50.614396114s) [seg_pos={0 0}, upload=449937]: Segment reached optimal threshold of 13 pieces. Dec 27 00:13:50.743502049 (512h44m50.614585114s) [noise=true , address=145.79.227.15:4875 , piece_id=G7MFJ3GU3I5YLGQQNSTTS6ZYMPZC6C4AGRW2TPF5GZFSPX2NO4ZQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 27 00:13:50.743512660 (512h44m50.614595705s) [noise=true , address=145.79.227.15:4875 , piece_id=G7MFJ3GU3I5YLGQQNSTTS6ZYMPZC6C4AGRW2TPF5GZFSPX2NO4ZQ, seg_pos={0 0}, upload=449937]: Long tail cancellation due to optimal threshold Dec 27 00:13:50.743680720 (512h44m50.614763765s) [noise=true , address=45.91.116.11:4119 , piece_id=KHDUBLEBIX74USWMPWOSDS5I2S2L5XFOUH44V7T6C73DT7MAN3QA, seg_pos={0 0}, upload=449937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 27 00:13:50.743687723 (512h44m50.614770758s) [noise=true , address=45.91.116.11:4119 , piece_id=KHDUBLEBIX74USWMPWOSDS5I2S2L5XFOUH44V7T6C73DT7MAN3QA, seg_pos={0 0}, upload=449937]: Long tail cancellation due to optimal threshold Dec 27 00:13:50.743715847 (512h44m50.614798892s) [noise=true , address=45.91.116.10:4084 , piece_id=HNUW4H7NRS4RME2BEZBSO3XCGPG4DFUDNPRBRQ5AV7IRLNOF24LQ, seg_pos={0 0}, upload=449937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: read tcp 188.166.225.170:39470->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 00:13:50.743734261 (512h44m50.614817306s) [noise=true , address=45.91.116.10:4084 , piece_id=HNUW4H7NRS4RME2BEZBSO3XCGPG4DFUDNPRBRQ5AV7IRLNOF24LQ, seg_pos={0 0}, upload=449937]: Long tail cancellation due to optimal threshold Dec 27 00:13:50.743761203 (512h44m50.614844238s) [seg_pos={0 0}, upload=449937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:13:50.743932890 (512h44m50.615015925s) [upload=449937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:13:50.743939793 (512h44m50.615022838s) [upload=449937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:13:51.035407432 (512h44m50.906490468s) [upload=449937]: Done uploading segments... Dec 27 00:13:51.035427089 (512h44m50.906510124s) [upload=449937]: Upload finished. err: <nil> Duration: 30.00371839s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 255.840181ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:13:30 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 255.864346ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X..XX.XX..XXXX..X.XX...X.........X.........X.......X.X] duration: 30.004391673s trace: https://jaeger.dev.storj.io/trace/31720efab4596045 FAIL (2025-12-27 00:13:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 5OTYICRHAXRK75KXZDBB3MGTERJ2GYSJAJRYFT2PRGGDQ34B2EXA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: M6X35Z4WZNHVRHM5TKFZQLHBYSOFYVEBKZAE7JRTQBAG6AGHZW6Q) context deadline exceeded ======LOGS====== Dec 27 00:12:45.043074119 (512h43m44.914157154s) [upload=449923]: Starting upload Dec 27 00:12:45.043163259 (512h43m44.914246294s) [upload=449923]: Done starting upload Dec 27 00:12:45.043219306 (512h43m44.914302341s) [upload=449923]: Uploading segments... Dec 27 00:12:45.043483008 (512h43m44.914566063s) [upload=449923]: Got next segment. Inline: false Dec 27 00:12:45.043525889 (512h43m44.914608934s) [upload=449923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 00:12:45.043532492 (512h43m44.914615537s) [upload=449923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 00:12:45.288828635 (512h43m45.15991168s) [seg_pos={0 0}, upload=449923]: Begin upload segment... Dec 27 00:12:45.288869042 (512h43m45.159952077s) [seg_pos={0 0}, upload=449923]: Done begin upload segment. Dec 27 00:12:45.288893569 (512h43m45.159976604s) [noise=true , address=77.90.31.14:4664 , piece_id=RCMDHUIJ5FQA2HE3QDGP4LKLZPHISTA6NHGL4MGR7UK7HYLBCVLA, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.289052411 (512h43m45.160135446s) [noise=true , address=145.79.227.45:4229 , piece_id=ROP7VPR6BQRAKLG2KRCAX6SYPHG5E6APV25NQI5SUSFXQEWFRRKQ, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.289478492 (512h43m45.160561527s) [noise=true , address=145.79.227.50:4375 , piece_id=6LNUQDGHYYWWA5SWM6WJBJQIH6MRPF2LS34UYXEKYDD6RCCZDWXA, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.289776250 (512h43m45.160859285s) [noise=true , address=77.90.31.17:4773 , piece_id=ESIDFEAPKDLVK4BQHORSVTBWM6FW3IYOTUJZBYLFX3STY4ZEJS4Q, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.290297272 (512h43m45.161380317s) [noise=true , address=145.79.227.38:4977 , piece_id=EI5MBSYQ3QB3WNG4NSXG6PBDC3LMEWKJHERHRZ5FIZBCDTDUQCDA, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.290608494 (512h43m45.161691529s) [noise=true , address=145.79.227.42:4122 , piece_id=BEM7H2JN6FIDSFNLU73IIIPE3V7G4CZDZIPZC2G5EQ4MKGIG4B7A, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.290952640 (512h43m45.162035675s) [noise=true , address=145.79.227.47:4266 , piece_id=OI5WUQD2N5BXZM52SONUDQVIRXDKOS2NHP446MSLEOJEAYCJ2HAQ, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.291298649 (512h43m45.162381694s) [noise=true , address=145.79.227.52:4447 , piece_id=JKPYDM4BOWY55B4SS22RK6CMBOB7VXLKF36GVYBFG6GTDYKGDS6Q, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.291670125 (512h43m45.162753181s) [noise=true , address=145.79.227.15:4873 , piece_id=WCSJFX2QCKLVUDGRHYQ3CE6YVJLTRVIGBFEHLSV4E75RXYAVUK5A, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.292734783 (512h43m45.163817818s) [noise=true , address=145.79.227.6:4258 , piece_id=2CANIJRFZLBOVA2RBUYJWCAQ7N3HP6DI72OFWKJUHOJSJEAVL6CA, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.293011280 (512h43m45.164094315s) [noise=true , address=45.91.116.8:4047 , piece_id=M6X35Z4WZNHVRHM5TKFZQLHBYSOFYVEBKZAE7JRTQBAG6AGHZW6Q, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.294238065 (512h43m45.165321101s) [noise=true , address=77.90.31.15:4700 , piece_id=JOBQO2DWCWMFFB4GBG5RHIJQLI7ZGVVPTVDLQIOZWCYNUO2IXJ2Q, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.295301170 (512h43m45.166384215s) [noise=true , address=145.79.227.37:4567 , piece_id=NI6ESD6JYOBTMTZJDZHTINZ7P6BABWXISNT64XFVIARJIF7OMYUA, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.297197411 (512h43m45.168280456s) [noise=true , address=45.91.116.11:4117 , piece_id=5OTYICRHAXRK75KXZDBB3MGTERJ2GYSJAJRYFT2PRGGDQ34B2EXA, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.297643250 (512h43m45.168726295s) [noise=true , address=45.91.116.14:4190 , piece_id=S5EJKUICGU5GBBH2FLH5NY6OVUQNXCQQWTOZ2EY5GDEINZN5BKWQ, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:45.298777028 (512h43m45.169860063s) [noise=true , address=45.91.116.6:4973 , piece_id=FWRLWUTLCDKSRFFDBLPG5FD7D655QJXBHBYL5H2NSCLPPBSTLCLQ, seg_pos={0 0}, upload=449923]: Uploading piece... Dec 27 00:12:46.996583846 (512h43m46.867666891s) [upload=449923]: Next returned nil segment Dec 27 00:12:46.996597993 (512h43m46.867681028s) [upload=449923]: Waiting for error group managing segments... Dec 27 00:12:47.289147942 (512h43m47.160230977s) [noise=true , address=145.79.227.50:4375 , piece_id=6LNUQDGHYYWWA5SWM6WJBJQIH6MRPF2LS34UYXEKYDD6RCCZDWXA, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.290647307 (512h43m47.161730352s) [noise=true , address=145.79.227.37:4567 , piece_id=NI6ESD6JYOBTMTZJDZHTINZ7P6BABWXISNT64XFVIARJIF7OMYUA, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.298974754 (512h43m47.170057799s) [noise=true , address=145.79.227.45:4229 , piece_id=ROP7VPR6BQRAKLG2KRCAX6SYPHG5E6APV25NQI5SUSFXQEWFRRKQ, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.301838988 (512h43m47.172922033s) [noise=true , address=145.79.227.42:4122 , piece_id=BEM7H2JN6FIDSFNLU73IIIPE3V7G4CZDZIPZC2G5EQ4MKGIG4B7A, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.303552952 (512h43m47.174635987s) [noise=true , address=145.79.227.47:4266 , piece_id=OI5WUQD2N5BXZM52SONUDQVIRXDKOS2NHP446MSLEOJEAYCJ2HAQ, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.304224971 (512h43m47.175308006s) [noise=true , address=145.79.227.38:4977 , piece_id=EI5MBSYQ3QB3WNG4NSXG6PBDC3LMEWKJHERHRZ5FIZBCDTDUQCDA, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.311075245 (512h43m47.18215828s) [noise=true , address=145.79.227.15:4873 , piece_id=WCSJFX2QCKLVUDGRHYQ3CE6YVJLTRVIGBFEHLSV4E75RXYAVUK5A, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.313030579 (512h43m47.184113624s) [noise=true , address=145.79.227.52:4447 , piece_id=JKPYDM4BOWY55B4SS22RK6CMBOB7VXLKF36GVYBFG6GTDYKGDS6Q, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.329853642 (512h43m47.200936677s) [noise=true , address=145.79.227.6:4258 , piece_id=2CANIJRFZLBOVA2RBUYJWCAQ7N3HP6DI72OFWKJUHOJSJEAVL6CA, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.470985434 (512h43m47.342068479s) [noise=true , address=77.90.31.14:4664 , piece_id=RCMDHUIJ5FQA2HE3QDGP4LKLZPHISTA6NHGL4MGR7UK7HYLBCVLA, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.512622165 (512h43m47.38370521s) [noise=true , address=45.91.116.8:4047 , piece_id=M6X35Z4WZNHVRHM5TKFZQLHBYSOFYVEBKZAE7JRTQBAG6AGHZW6Q, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.518844503 (512h43m47.389927538s) [noise=true , address=45.91.116.11:4117 , piece_id=5OTYICRHAXRK75KXZDBB3MGTERJ2GYSJAJRYFT2PRGGDQ34B2EXA, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.561346532 (512h43m47.432429567s) [noise=true , address=77.90.31.15:4700 , piece_id=JOBQO2DWCWMFFB4GBG5RHIJQLI7ZGVVPTVDLQIOZWCYNUO2IXJ2Q, seg_pos={0 0}, upload=449923]: Done uploading piece. err: <nil> Dec 27 00:12:47.561373252 (512h43m47.432456287s) [seg_pos={0 0}, upload=449923]: Segment reached optimal threshold of 13 pieces. Dec 27 00:12:47.561637526 (512h43m47.432720571s) [noise=true , address=77.90.31.17:4773 , piece_id=ESIDFEAPKDLVK4BQHORSVTBWM6FW3IYOTUJZBYLFX3STY4ZEJS4Q, seg_pos={0 0}, upload=449923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 27 00:12:47.561649097 (512h43m47.432732142s) [noise=true , address=77.90.31.17:4773 , piece_id=ESIDFEAPKDLVK4BQHORSVTBWM6FW3IYOTUJZBYLFX3STY4ZEJS4Q, seg_pos={0 0}, upload=449923]: Long tail cancellation due to optimal threshold Dec 27 00:12:47.561837306 (512h43m47.432920361s) [noise=true , address=45.91.116.6:4973 , piece_id=FWRLWUTLCDKSRFFDBLPG5FD7D655QJXBHBYL5H2NSCLPPBSTLCLQ, seg_pos={0 0}, upload=449923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 27 00:12:47.561856983 (512h43m47.432940028s) [noise=true , address=45.91.116.6:4973 , piece_id=FWRLWUTLCDKSRFFDBLPG5FD7D655QJXBHBYL5H2NSCLPPBSTLCLQ, seg_pos={0 0}, upload=449923]: Long tail cancellation due to optimal threshold Dec 27 00:12:47.562187072 (512h43m47.433270127s) [noise=true , address=45.91.116.14:4190 , piece_id=S5EJKUICGU5GBBH2FLH5NY6OVUQNXCQQWTOZ2EY5GDEINZN5BKWQ, seg_pos={0 0}, upload=449923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 00:12:47.562198894 (512h43m47.43328194s) [noise=true , address=45.91.116.14:4190 , piece_id=S5EJKUICGU5GBBH2FLH5NY6OVUQNXCQQWTOZ2EY5GDEINZN5BKWQ, seg_pos={0 0}, upload=449923]: Long tail cancellation due to optimal threshold Dec 27 00:12:47.562231497 (512h43m47.433314542s) [seg_pos={0 0}, upload=449923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 00:12:47.562327379 (512h43m47.433410425s) [upload=449923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 00:12:47.562334573 (512h43m47.433417618s) [upload=449923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 00:12:47.852313718 (512h43m47.723396763s) [upload=449923]: Done uploading segments... Dec 27 00:12:47.852337524 (512h43m47.723420559s) [upload=449923]: Upload finished. err: <nil> Duration: 30.004412201s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 258.484605ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:12:44 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 258.509753ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 257.419327ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/67e64704eddcd2cd FAIL (2025-12-27 00:12:07 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 257.52628ms edge linksharing bgp eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)