Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
34.37589288s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:58:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 34.37591909s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X.X.XXXXX.X.] duration:
6.41468974s trace:
https://jaeger.dev.storj.io/trace/7973da8c9f0d0a39
FAIL (2025-12-27 00:58:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: NYEDWVSGSQMIVWYUSK63BV6OBRXM43MYR25A54DDPD6ITJWIR6QA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: OS3PTXRGKQB236UHIIEVIRNOT2RLFJWPAT5MO56ZPOU5G6MXKVWA) context deadline exceeded
======LOGS======
Dec 27 00:57:26.555841154 (513h28m26.426924189s) [upload=450573]: Starting upload
Dec 27 00:57:26.555934292 (513h28m26.427017347s) [upload=450573]: Done starting upload
Dec 27 00:57:26.556016658 (513h28m26.427099693s) [upload=450573]: Uploading segments...
Dec 27 00:57:26.556224514 (513h28m26.427307559s) [upload=450573]: Got next segment. Inline: false
Dec 27 00:57:26.556264460 (513h28m26.427347525s) [upload=450573]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:57:26.556270531 (513h28m26.427353576s) [upload=450573]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:57:26.800957196 (513h28m26.672040231s) [seg_pos={0 0}, upload=450573]: Begin upload segment...
Dec 27 00:57:26.801012611 (513h28m26.672095676s) [seg_pos={0 0}, upload=450573]: Done begin upload segment.
Dec 27 00:57:26.801048960 (513h28m26.672132005s) [noise=true , address=45.91.116.14:4189 , piece_id=ADHYEAMGL7FMAQ5BXUM5KHBE3B7NDVMYGQRWF2QJWO6VOXVYT73A, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.801082774 (513h28m26.672165809s) [noise=true , address=45.91.116.7:4009 , piece_id=OS3PTXRGKQB236UHIIEVIRNOT2RLFJWPAT5MO56ZPOU5G6MXKVWA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.801741859 (513h28m26.672824905s) [noise=true , address=145.79.227.39:4015 , piece_id=RQJ5FFF2O6MKNYR4WRMCCTJQGO4MPPLX3JZKN432FKPEJFWJWTFQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.802291617 (513h28m26.673374662s) [noise=true , address=45.91.116.11:4120 , piece_id=GBA7XQ3BSCLILYXP7GGMDPGIJ7VZ273GUFR5JXKOWGOXQSERUE2A, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.802660168 (513h28m26.673743213s) [noise=true , address=145.79.227.36:4518 , piece_id=UYCVMBT2EI3OZO3Q6CQ724OA3KRWORHFDU326ZWWPS73SBXTA4TQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.803034972 (513h28m26.674118027s) [noise=true , address=45.91.116.9:4263 , piece_id=FUBM4NQW2L5WWUQMYIA5JDAMPM53EOEVV5DMTNS5JIWRCBUJM6YA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.803447327 (513h28m26.674530372s) [noise=true , address=145.79.227.47:4267 , piece_id=ADYTMVJ5CVJTNACYUUMXM6N4AILVCHEBWM4BILRPHEN5RIPFQRJQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.803821219 (513h28m26.674904264s) [noise=true , address=145.79.227.15:4875 , piece_id=5L7CPJBJUGDQG6RB74JUBDIH4SEQHGWQPSN74NC25KNYAAYBM7ZQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.804260715 (513h28m26.675343751s) [noise=true , address=145.79.227.48:4302 , piece_id=LV32XJIGXCVEC6DDETDJHUMDKR4M6N7KHFGFA55ASAQE4YILXTXQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.806252519 (513h28m26.677335564s) [noise=true , address=145.79.227.51:4411 , piece_id=BJAVTXCHNMPHVKEX4TOBTUI5UOZS6VB3WMZNQYMII6AJS6LDZUMA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.808085669 (513h28m26.679168704s) [noise=true , address=77.90.31.10:4517 , piece_id=NYEDWVSGSQMIVWYUSK63BV6OBRXM43MYR25A54DDPD6ITJWIR6QA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.809974025 (513h28m26.681057071s) [noise=true , address=145.79.227.46:4602 , piece_id=QPH7BXCJZMQL4OLGWH3PAU47R72JQ36KFNAQKCJG7XZYRLTRZEZA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.811700643 (513h28m26.682783688s) [noise=true , address=145.79.227.52:4445 , piece_id=52KHLDKOBAEKHTPR6LK2YLKTDPQQ77W3UDFKW7LMOI3VACXITYRA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.813228652 (513h28m26.684311697s) [noise=true , address=77.90.31.15:4697 , piece_id=GK5TNZVT6WXXNBDPAYER5GF4RSE7AHDGE3D5YPYFR2NRE344P4BQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.814812698 (513h28m26.685895743s) [noise=true , address=77.90.31.14:4664 , piece_id=77N4GMJNEGSPRRFPTYLQRO7PWNZ5D5RFAZQ6R6T2SW3O6UM22YMQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.816674253 (513h28m26.687757298s) [noise=true , address=77.90.31.18:4805 , piece_id=36FRNHXKJNU5BYGV4SV36IBBE2G2K3HFY566UVP3BMKTS4GDJAQA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:28.521285593 (513h28m28.392368628s) [upload=450573]: Next returned nil segment
Dec 27 00:57:28.521305842 (513h28m28.392388887s) [upload=450573]: Waiting for error group managing segments...
Dec 27 00:57:28.800203429 (513h28m28.671286474s) [noise=true , address=145.79.227.47:4267 , piece_id=ADYTMVJ5CVJTNACYUUMXM6N4AILVCHEBWM4BILRPHEN5RIPFQRJQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.808565671 (513h28m28.679648706s) [noise=true , address=145.79.227.36:4518 , piece_id=UYCVMBT2EI3OZO3Q6CQ724OA3KRWORHFDU326ZWWPS73SBXTA4TQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.822046551 (513h28m28.693129596s) [noise=true , address=145.79.227.51:4411 , piece_id=BJAVTXCHNMPHVKEX4TOBTUI5UOZS6VB3WMZNQYMII6AJS6LDZUMA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.823933043 (513h28m28.695016078s) [noise=true , address=145.79.227.15:4875 , piece_id=5L7CPJBJUGDQG6RB74JUBDIH4SEQHGWQPSN74NC25KNYAAYBM7ZQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.825375800 (513h28m28.696458835s) [noise=true , address=145.79.227.52:4445 , piece_id=52KHLDKOBAEKHTPR6LK2YLKTDPQQ77W3UDFKW7LMOI3VACXITYRA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.830406690 (513h28m28.701489735s) [noise=true , address=145.79.227.48:4302 , piece_id=LV32XJIGXCVEC6DDETDJHUMDKR4M6N7KHFGFA55ASAQE4YILXTXQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.830778227 (513h28m28.701861272s) [noise=true , address=145.79.227.39:4015 , piece_id=RQJ5FFF2O6MKNYR4WRMCCTJQGO4MPPLX3JZKN432FKPEJFWJWTFQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.852383336 (513h28m28.723466381s) [noise=true , address=145.79.227.46:4602 , piece_id=QPH7BXCJZMQL4OLGWH3PAU47R72JQ36KFNAQKCJG7XZYRLTRZEZA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.004617850 (513h28m28.875700885s) [noise=true , address=77.90.31.14:4664 , piece_id=77N4GMJNEGSPRRFPTYLQRO7PWNZ5D5RFAZQ6R6T2SW3O6UM22YMQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.058337569 (513h28m28.929420604s) [noise=true , address=45.91.116.11:4120 , piece_id=GBA7XQ3BSCLILYXP7GGMDPGIJ7VZ273GUFR5JXKOWGOXQSERUE2A, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.070385680 (513h28m28.941468725s) [noise=true , address=77.90.31.10:4517 , piece_id=NYEDWVSGSQMIVWYUSK63BV6OBRXM43MYR25A54DDPD6ITJWIR6QA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.157449802 (513h28m29.028532847s) [noise=true , address=45.91.116.14:4189 , piece_id=ADHYEAMGL7FMAQ5BXUM5KHBE3B7NDVMYGQRWF2QJWO6VOXVYT73A, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.164776302 (513h28m29.035859337s) [noise=true , address=45.91.116.7:4009 , piece_id=OS3PTXRGKQB236UHIIEVIRNOT2RLFJWPAT5MO56ZPOU5G6MXKVWA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.164798264 (513h28m29.035881299s) [seg_pos={0 0}, upload=450573]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:57:29.164954030 (513h28m29.036037065s) [noise=true , address=77.90.31.15:4697 , piece_id=GK5TNZVT6WXXNBDPAYER5GF4RSE7AHDGE3D5YPYFR2NRE344P4BQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 27 00:57:29.164960213 (513h28m29.036043268s) [noise=true , address=77.90.31.15:4697 , piece_id=GK5TNZVT6WXXNBDPAYER5GF4RSE7AHDGE3D5YPYFR2NRE344P4BQ, seg_pos={0 0}, upload=450573]: Long tail cancellation due to optimal threshold
Dec 27 00:57:29.165062758 (513h28m29.036145793s) [noise=true , address=77.90.31.18:4805 , piece_id=36FRNHXKJNU5BYGV4SV36IBBE2G2K3HFY566UVP3BMKTS4GDJAQA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 27 00:57:29.165070463 (513h28m29.036153508s) [noise=true , address=77.90.31.18:4805 , piece_id=36FRNHXKJNU5BYGV4SV36IBBE2G2K3HFY566UVP3BMKTS4GDJAQA, seg_pos={0 0}, upload=450573]: Long tail cancellation due to optimal threshold
Dec 27 00:57:29.165114035 (513h28m29.03619708s) [noise=true , address=45.91.116.9:4263 , piece_id=FUBM4NQW2L5WWUQMYIA5JDAMPM53EOEVV5DMTNS5JIWRCBUJM6YA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 27 00:57:29.165120077 (513h28m29.036203112s) [noise=true , address=45.91.116.9:4263 , piece_id=FUBM4NQW2L5WWUQMYIA5JDAMPM53EOEVV5DMTNS5JIWRCBUJM6YA, seg_pos={0 0}, upload=450573]: Long tail cancellation due to optimal threshold
Dec 27 00:57:29.165153160 (513h28m29.036236195s) [seg_pos={0 0}, upload=450573]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:57:29.165255184 (513h28m29.036338229s) [upload=450573]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:57:29.165260504 (513h28m29.036343549s) [upload=450573]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:57:29.449156780 (513h28m29.320239825s) [upload=450573]: Done uploading segments...
Dec 27 00:57:29.449183430 (513h28m29.320266475s) [upload=450573]: Upload finished. err: <nil>
Duration: 6.414720086s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X.X.XXXXX.X] duration:
30.003801914s trace:
https://jaeger.dev.storj.io/trace/7973da8c9f0d0a39
FAIL (2025-12-27 00:57:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: NYEDWVSGSQMIVWYUSK63BV6OBRXM43MYR25A54DDPD6ITJWIR6QA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: OS3PTXRGKQB236UHIIEVIRNOT2RLFJWPAT5MO56ZPOU5G6MXKVWA) context deadline exceeded
======LOGS======
Dec 27 00:57:26.555841154 (513h28m26.426924189s) [upload=450573]: Starting upload
Dec 27 00:57:26.555934292 (513h28m26.427017347s) [upload=450573]: Done starting upload
Dec 27 00:57:26.556016658 (513h28m26.427099693s) [upload=450573]: Uploading segments...
Dec 27 00:57:26.556224514 (513h28m26.427307559s) [upload=450573]: Got next segment. Inline: false
Dec 27 00:57:26.556264460 (513h28m26.427347525s) [upload=450573]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:57:26.556270531 (513h28m26.427353576s) [upload=450573]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:57:26.800957196 (513h28m26.672040231s) [seg_pos={0 0}, upload=450573]: Begin upload segment...
Dec 27 00:57:26.801012611 (513h28m26.672095676s) [seg_pos={0 0}, upload=450573]: Done begin upload segment.
Dec 27 00:57:26.801048960 (513h28m26.672132005s) [noise=true , address=45.91.116.14:4189 , piece_id=ADHYEAMGL7FMAQ5BXUM5KHBE3B7NDVMYGQRWF2QJWO6VOXVYT73A, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.801082774 (513h28m26.672165809s) [noise=true , address=45.91.116.7:4009 , piece_id=OS3PTXRGKQB236UHIIEVIRNOT2RLFJWPAT5MO56ZPOU5G6MXKVWA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.801741859 (513h28m26.672824905s) [noise=true , address=145.79.227.39:4015 , piece_id=RQJ5FFF2O6MKNYR4WRMCCTJQGO4MPPLX3JZKN432FKPEJFWJWTFQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.802291617 (513h28m26.673374662s) [noise=true , address=45.91.116.11:4120 , piece_id=GBA7XQ3BSCLILYXP7GGMDPGIJ7VZ273GUFR5JXKOWGOXQSERUE2A, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.802660168 (513h28m26.673743213s) [noise=true , address=145.79.227.36:4518 , piece_id=UYCVMBT2EI3OZO3Q6CQ724OA3KRWORHFDU326ZWWPS73SBXTA4TQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.803034972 (513h28m26.674118027s) [noise=true , address=45.91.116.9:4263 , piece_id=FUBM4NQW2L5WWUQMYIA5JDAMPM53EOEVV5DMTNS5JIWRCBUJM6YA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.803447327 (513h28m26.674530372s) [noise=true , address=145.79.227.47:4267 , piece_id=ADYTMVJ5CVJTNACYUUMXM6N4AILVCHEBWM4BILRPHEN5RIPFQRJQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.803821219 (513h28m26.674904264s) [noise=true , address=145.79.227.15:4875 , piece_id=5L7CPJBJUGDQG6RB74JUBDIH4SEQHGWQPSN74NC25KNYAAYBM7ZQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.804260715 (513h28m26.675343751s) [noise=true , address=145.79.227.48:4302 , piece_id=LV32XJIGXCVEC6DDETDJHUMDKR4M6N7KHFGFA55ASAQE4YILXTXQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.806252519 (513h28m26.677335564s) [noise=true , address=145.79.227.51:4411 , piece_id=BJAVTXCHNMPHVKEX4TOBTUI5UOZS6VB3WMZNQYMII6AJS6LDZUMA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.808085669 (513h28m26.679168704s) [noise=true , address=77.90.31.10:4517 , piece_id=NYEDWVSGSQMIVWYUSK63BV6OBRXM43MYR25A54DDPD6ITJWIR6QA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.809974025 (513h28m26.681057071s) [noise=true , address=145.79.227.46:4602 , piece_id=QPH7BXCJZMQL4OLGWH3PAU47R72JQ36KFNAQKCJG7XZYRLTRZEZA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.811700643 (513h28m26.682783688s) [noise=true , address=145.79.227.52:4445 , piece_id=52KHLDKOBAEKHTPR6LK2YLKTDPQQ77W3UDFKW7LMOI3VACXITYRA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.813228652 (513h28m26.684311697s) [noise=true , address=77.90.31.15:4697 , piece_id=GK5TNZVT6WXXNBDPAYER5GF4RSE7AHDGE3D5YPYFR2NRE344P4BQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.814812698 (513h28m26.685895743s) [noise=true , address=77.90.31.14:4664 , piece_id=77N4GMJNEGSPRRFPTYLQRO7PWNZ5D5RFAZQ6R6T2SW3O6UM22YMQ, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:26.816674253 (513h28m26.687757298s) [noise=true , address=77.90.31.18:4805 , piece_id=36FRNHXKJNU5BYGV4SV36IBBE2G2K3HFY566UVP3BMKTS4GDJAQA, seg_pos={0 0}, upload=450573]: Uploading piece...
Dec 27 00:57:28.521285593 (513h28m28.392368628s) [upload=450573]: Next returned nil segment
Dec 27 00:57:28.521305842 (513h28m28.392388887s) [upload=450573]: Waiting for error group managing segments...
Dec 27 00:57:28.800203429 (513h28m28.671286474s) [noise=true , address=145.79.227.47:4267 , piece_id=ADYTMVJ5CVJTNACYUUMXM6N4AILVCHEBWM4BILRPHEN5RIPFQRJQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.808565671 (513h28m28.679648706s) [noise=true , address=145.79.227.36:4518 , piece_id=UYCVMBT2EI3OZO3Q6CQ724OA3KRWORHFDU326ZWWPS73SBXTA4TQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.822046551 (513h28m28.693129596s) [noise=true , address=145.79.227.51:4411 , piece_id=BJAVTXCHNMPHVKEX4TOBTUI5UOZS6VB3WMZNQYMII6AJS6LDZUMA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.823933043 (513h28m28.695016078s) [noise=true , address=145.79.227.15:4875 , piece_id=5L7CPJBJUGDQG6RB74JUBDIH4SEQHGWQPSN74NC25KNYAAYBM7ZQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.825375800 (513h28m28.696458835s) [noise=true , address=145.79.227.52:4445 , piece_id=52KHLDKOBAEKHTPR6LK2YLKTDPQQ77W3UDFKW7LMOI3VACXITYRA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.830406690 (513h28m28.701489735s) [noise=true , address=145.79.227.48:4302 , piece_id=LV32XJIGXCVEC6DDETDJHUMDKR4M6N7KHFGFA55ASAQE4YILXTXQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.830778227 (513h28m28.701861272s) [noise=true , address=145.79.227.39:4015 , piece_id=RQJ5FFF2O6MKNYR4WRMCCTJQGO4MPPLX3JZKN432FKPEJFWJWTFQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:28.852383336 (513h28m28.723466381s) [noise=true , address=145.79.227.46:4602 , piece_id=QPH7BXCJZMQL4OLGWH3PAU47R72JQ36KFNAQKCJG7XZYRLTRZEZA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.004617850 (513h28m28.875700885s) [noise=true , address=77.90.31.14:4664 , piece_id=77N4GMJNEGSPRRFPTYLQRO7PWNZ5D5RFAZQ6R6T2SW3O6UM22YMQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.058337569 (513h28m28.929420604s) [noise=true , address=45.91.116.11:4120 , piece_id=GBA7XQ3BSCLILYXP7GGMDPGIJ7VZ273GUFR5JXKOWGOXQSERUE2A, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.070385680 (513h28m28.941468725s) [noise=true , address=77.90.31.10:4517 , piece_id=NYEDWVSGSQMIVWYUSK63BV6OBRXM43MYR25A54DDPD6ITJWIR6QA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.157449802 (513h28m29.028532847s) [noise=true , address=45.91.116.14:4189 , piece_id=ADHYEAMGL7FMAQ5BXUM5KHBE3B7NDVMYGQRWF2QJWO6VOXVYT73A, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.164776302 (513h28m29.035859337s) [noise=true , address=45.91.116.7:4009 , piece_id=OS3PTXRGKQB236UHIIEVIRNOT2RLFJWPAT5MO56ZPOU5G6MXKVWA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: <nil>
Dec 27 00:57:29.164798264 (513h28m29.035881299s) [seg_pos={0 0}, upload=450573]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:57:29.164954030 (513h28m29.036037065s) [noise=true , address=77.90.31.15:4697 , piece_id=GK5TNZVT6WXXNBDPAYER5GF4RSE7AHDGE3D5YPYFR2NRE344P4BQ, seg_pos={0 0}, upload=450573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 27 00:57:29.164960213 (513h28m29.036043268s) [noise=true , address=77.90.31.15:4697 , piece_id=GK5TNZVT6WXXNBDPAYER5GF4RSE7AHDGE3D5YPYFR2NRE344P4BQ, seg_pos={0 0}, upload=450573]: Long tail cancellation due to optimal threshold
Dec 27 00:57:29.165062758 (513h28m29.036145793s) [noise=true , address=77.90.31.18:4805 , piece_id=36FRNHXKJNU5BYGV4SV36IBBE2G2K3HFY566UVP3BMKTS4GDJAQA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 27 00:57:29.165070463 (513h28m29.036153508s) [noise=true , address=77.90.31.18:4805 , piece_id=36FRNHXKJNU5BYGV4SV36IBBE2G2K3HFY566UVP3BMKTS4GDJAQA, seg_pos={0 0}, upload=450573]: Long tail cancellation due to optimal threshold
Dec 27 00:57:29.165114035 (513h28m29.03619708s) [noise=true , address=45.91.116.9:4263 , piece_id=FUBM4NQW2L5WWUQMYIA5JDAMPM53EOEVV5DMTNS5JIWRCBUJM6YA, seg_pos={0 0}, upload=450573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 27 00:57:29.165120077 (513h28m29.036203112s) [noise=true , address=45.91.116.9:4263 , piece_id=FUBM4NQW2L5WWUQMYIA5JDAMPM53EOEVV5DMTNS5JIWRCBUJM6YA, seg_pos={0 0}, upload=450573]: Long tail cancellation due to optimal threshold
Dec 27 00:57:29.165153160 (513h28m29.036236195s) [seg_pos={0 0}, upload=450573]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:57:29.165255184 (513h28m29.036338229s) [upload=450573]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:57:29.165260504 (513h28m29.036343549s) [upload=450573]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:57:29.449156780 (513h28m29.320239825s) [upload=450573]: Done uploading segments...
Dec 27 00:57:29.449183430 (513h28m29.320266475s) [upload=450573]: Upload finished. err: <nil>
Duration: 30.003863159s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
7.222816515s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:57:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 7.222842925s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [.X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X.X.XXXXX.] duration:
5.884522448s trace:
https://jaeger.dev.storj.io/trace/35f8a00499b0694e
FAIL (2025-12-27 00:57:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: KFU42A23RPS6TZOZDNRT6DPBOCCCTKWZRUD2WC4ZUKS5MMVVYKMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: NHDIPNWZ2SX3C5ERYR47Q3KDCSAAEA55VAPIXBAJGBMSKYYB5VNQ) context deadline exceeded
======LOGS======
Dec 27 00:56:33.927788351 (513h27m33.798871387s) [upload=450558]: Starting upload
Dec 27 00:56:33.927838136 (513h27m33.798921181s) [upload=450558]: Done starting upload
Dec 27 00:56:33.928031735 (513h27m33.79911477s) [upload=450558]: Uploading segments...
Dec 27 00:56:33.928068555 (513h27m33.79915159s) [upload=450558]: Got next segment. Inline: false
Dec 27 00:56:33.928092911 (513h27m33.799175956s) [upload=450558]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:56:33.928098541 (513h27m33.799181586s) [upload=450558]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:56:34.174030487 (513h27m34.045113532s) [seg_pos={0 0}, upload=450558]: Begin upload segment...
Dec 27 00:56:34.174096823 (513h27m34.045179858s) [seg_pos={0 0}, upload=450558]: Done begin upload segment.
Dec 27 00:56:34.174168059 (513h27m34.045251114s) [noise=true , address=45.91.116.5:4300 , piece_id=KFU42A23RPS6TZOZDNRT6DPBOCCCTKWZRUD2WC4ZUKS5MMVVYKMA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.174205160 (513h27m34.045288195s) [noise=true , address=145.79.227.5:4221 , piece_id=OACPNPTF42HLJ3IZZQ7V2MJX26SXV6J73WAGANUN4HPHVMFWXUYQ, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.174786626 (513h27m34.045869661s) [noise=true , address=45.91.116.12:4225 , piece_id=NHDIPNWZ2SX3C5ERYR47Q3KDCSAAEA55VAPIXBAJGBMSKYYB5VNQ, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.175192059 (513h27m34.046275094s) [noise=true , address=145.79.227.29:4941 , piece_id=QBVC4QPPS3WXESGRE5OVFYPRQT2IUMLDGUBXABC7AF6RN77EOBTA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.175577362 (513h27m34.046660397s) [noise=true , address=145.79.227.15:4874 , piece_id=ZW2UI253CRROCRLLJ45ZV7RCKO63CQN7NWRUGNZNBTGFSPTP5FTA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.176005157 (513h27m34.047088192s) [noise=true , address=145.79.227.38:4978 , piece_id=NF5X2SO4TEC5R274DB5F7PG27SKKDEGB2TKWKZHC33BVKWIXORNA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.176406391 (513h27m34.047489426s) [noise=true , address=145.79.227.6:4258 , piece_id=NDVV4X2IED52XMSIIHKJ3IQ25O5CCQZEDNRZANP6PFE773FUIGOA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.176807234 (513h27m34.047890269s) [noise=true , address=145.79.227.45:4229 , piece_id=HLBM7GOBJDVEZHY5XDFIOMUR6QIOZKBIEJGJXXDLATTEND5GHHBA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.177285144 (513h27m34.048368179s) [noise=true , address=145.79.227.36:4518 , piece_id=G5L5OEAB4Y63BWV3ELJSQI47FO64THL7KLXFSK2IG567KUQQGJLQ, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.178612612 (513h27m34.049695657s) [noise=true , address=145.79.227.19:4079 , piece_id=BPQTBALJDQNUAQLD4FFDP7K2LSUKWY74EUWXE3ZHXJUXCP3N324A, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.178850244 (513h27m34.04993328s) [noise=true , address=145.79.227.39:4014 , piece_id=UIH5AXUO7YAYWJC36ODWZ6CMEWH5UGEXEX6Z4DVRGZ67LABL44FA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.180281249 (513h27m34.051364294s) [noise=true , address=77.90.31.17:4770 , piece_id=Q46SB7DCMSUGTVTSGHF7CKRT4RHWKRCFLROIWVD4UVT7M7VEN5QA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.180363926 (513h27m34.051446971s) [noise=true , address=145.79.227.13:4803 , piece_id=5S75TV3QHKLRNATBPXUEG6JUXCG6UFFORLDI3W42UFKXQ5GZXK6A, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.181493117 (513h27m34.052576152s) [noise=true , address=145.79.227.47:4265 , piece_id=5JGM746GEVIL2SFESK7LBOH4MXII66PVF6PGAGA44U6QQNSHZFZA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.182207617 (513h27m34.053290652s) [noise=true , address=145.79.227.52:4447 , piece_id=ZAY6RYX6UNVEUW5WWZEWRJ2SH52QVQIBTPV2XAU64PKRU6XFLHHQ, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.182894656 (513h27m34.053977701s) [noise=true , address=145.79.227.48:4301 , piece_id=UHIQBGU25ZCTEZQRIDSC243A52AJIOBC2EE4UUTVOAJHX5MFSFKA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:36.034951419 (513h27m35.906034454s) [upload=450558]: Next returned nil segment
Dec 27 00:56:36.034969614 (513h27m35.906052649s) [upload=450558]: Waiting for error group managing segments...
Dec 27 00:56:36.347990616 (513h27m36.219073651s) [noise=true , address=145.79.227.5:4221 , piece_id=OACPNPTF42HLJ3IZZQ7V2MJX26SXV6J73WAGANUN4HPHVMFWXUYQ, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.355662444 (513h27m36.226745479s) [noise=true , address=145.79.227.45:4229 , piece_id=HLBM7GOBJDVEZHY5XDFIOMUR6QIOZKBIEJGJXXDLATTEND5GHHBA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.358096479 (513h27m36.229179524s) [noise=true , address=145.79.227.15:4874 , piece_id=ZW2UI253CRROCRLLJ45ZV7RCKO63CQN7NWRUGNZNBTGFSPTP5FTA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.359977050 (513h27m36.231060085s) [noise=true , address=145.79.227.52:4447 , piece_id=ZAY6RYX6UNVEUW5WWZEWRJ2SH52QVQIBTPV2XAU64PKRU6XFLHHQ, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.373113073 (513h27m36.244196118s) [noise=true , address=145.79.227.19:4079 , piece_id=BPQTBALJDQNUAQLD4FFDP7K2LSUKWY74EUWXE3ZHXJUXCP3N324A, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.387065871 (513h27m36.258148906s) [noise=true , address=145.79.227.36:4518 , piece_id=G5L5OEAB4Y63BWV3ELJSQI47FO64THL7KLXFSK2IG567KUQQGJLQ, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.387914778 (513h27m36.258997823s) [noise=true , address=145.79.227.6:4258 , piece_id=NDVV4X2IED52XMSIIHKJ3IQ25O5CCQZEDNRZANP6PFE773FUIGOA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.399672566 (513h27m36.270755601s) [noise=true , address=145.79.227.13:4803 , piece_id=5S75TV3QHKLRNATBPXUEG6JUXCG6UFFORLDI3W42UFKXQ5GZXK6A, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.426139071 (513h27m36.297222116s) [noise=true , address=45.91.116.5:4300 , piece_id=KFU42A23RPS6TZOZDNRT6DPBOCCCTKWZRUD2WC4ZUKS5MMVVYKMA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.494313273 (513h27m36.365396308s) [noise=true , address=77.90.31.17:4770 , piece_id=Q46SB7DCMSUGTVTSGHF7CKRT4RHWKRCFLROIWVD4UVT7M7VEN5QA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.591732612 (513h27m36.462815647s) [noise=true , address=45.91.116.12:4225 , piece_id=NHDIPNWZ2SX3C5ERYR47Q3KDCSAAEA55VAPIXBAJGBMSKYYB5VNQ, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.640303357 (513h27m36.511386402s) [noise=true , address=145.79.227.47:4265 , piece_id=5JGM746GEVIL2SFESK7LBOH4MXII66PVF6PGAGA44U6QQNSHZFZA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:37.070371981 (513h27m36.941455016s) [noise=true , address=145.79.227.38:4978 , piece_id=NF5X2SO4TEC5R274DB5F7PG27SKKDEGB2TKWKZHC33BVKWIXORNA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:37.070403962 (513h27m36.941487007s) [seg_pos={0 0}, upload=450558]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:56:37.070624612 (513h27m36.941707647s) [noise=true , address=145.79.227.39:4014 , piece_id=UIH5AXUO7YAYWJC36ODWZ6CMEWH5UGEXEX6Z4DVRGZ67LABL44FA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
Dec 27 00:56:37.070631735 (513h27m36.94171477s) [noise=true , address=145.79.227.39:4014 , piece_id=UIH5AXUO7YAYWJC36ODWZ6CMEWH5UGEXEX6Z4DVRGZ67LABL44FA, seg_pos={0 0}, upload=450558]: Long tail cancellation due to optimal threshold
Dec 27 00:56:37.070870079 (513h27m36.941953114s) [noise=true , address=145.79.227.29:4941 , piece_id=QBVC4QPPS3WXESGRE5OVFYPRQT2IUMLDGUBXABC7AF6RN77EOBTA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 27 00:56:37.070877393 (513h27m36.941960428s) [noise=true , address=145.79.227.29:4941 , piece_id=QBVC4QPPS3WXESGRE5OVFYPRQT2IUMLDGUBXABC7AF6RN77EOBTA, seg_pos={0 0}, upload=450558]: Long tail cancellation due to optimal threshold
Dec 27 00:56:37.071004305 (513h27m36.94208734s) [noise=true , address=145.79.227.48:4301 , piece_id=UHIQBGU25ZCTEZQRIDSC243A52AJIOBC2EE4UUTVOAJHX5MFSFKA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF
Dec 27 00:56:37.071011268 (513h27m36.942094313s) [noise=true , address=145.79.227.48:4301 , piece_id=UHIQBGU25ZCTEZQRIDSC243A52AJIOBC2EE4UUTVOAJHX5MFSFKA, seg_pos={0 0}, upload=450558]: Long tail cancellation due to optimal threshold
Dec 27 00:56:37.071029664 (513h27m36.942112699s) [seg_pos={0 0}, upload=450558]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:56:37.071110818 (513h27m36.942193853s) [upload=450558]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:56:37.071116418 (513h27m36.942199463s) [upload=450558]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:56:37.365768463 (513h27m37.236851498s) [upload=450558]: Done uploading segments...
Dec 27 00:56:37.365786918 (513h27m37.236869953s) [upload=450558]: Upload finished. err: <nil>
Duration: 5.884555942s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
22.218244327s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:57:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 22.218277651s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [..X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X.X.XXXXX] duration:
30.002756416s trace:
https://jaeger.dev.storj.io/trace/35f8a00499b0694e
FAIL (2025-12-27 00:57:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: KFU42A23RPS6TZOZDNRT6DPBOCCCTKWZRUD2WC4ZUKS5MMVVYKMA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: NHDIPNWZ2SX3C5ERYR47Q3KDCSAAEA55VAPIXBAJGBMSKYYB5VNQ) context deadline exceeded
======LOGS======
Dec 27 00:56:33.927788351 (513h27m33.798871387s) [upload=450558]: Starting upload
Dec 27 00:56:33.927838136 (513h27m33.798921181s) [upload=450558]: Done starting upload
Dec 27 00:56:33.928031735 (513h27m33.79911477s) [upload=450558]: Uploading segments...
Dec 27 00:56:33.928068555 (513h27m33.79915159s) [upload=450558]: Got next segment. Inline: false
Dec 27 00:56:33.928092911 (513h27m33.799175956s) [upload=450558]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:56:33.928098541 (513h27m33.799181586s) [upload=450558]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:56:34.174030487 (513h27m34.045113532s) [seg_pos={0 0}, upload=450558]: Begin upload segment...
Dec 27 00:56:34.174096823 (513h27m34.045179858s) [seg_pos={0 0}, upload=450558]: Done begin upload segment.
Dec 27 00:56:34.174168059 (513h27m34.045251114s) [noise=true , address=45.91.116.5:4300 , piece_id=KFU42A23RPS6TZOZDNRT6DPBOCCCTKWZRUD2WC4ZUKS5MMVVYKMA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.174205160 (513h27m34.045288195s) [noise=true , address=145.79.227.5:4221 , piece_id=OACPNPTF42HLJ3IZZQ7V2MJX26SXV6J73WAGANUN4HPHVMFWXUYQ, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.174786626 (513h27m34.045869661s) [noise=true , address=45.91.116.12:4225 , piece_id=NHDIPNWZ2SX3C5ERYR47Q3KDCSAAEA55VAPIXBAJGBMSKYYB5VNQ, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.175192059 (513h27m34.046275094s) [noise=true , address=145.79.227.29:4941 , piece_id=QBVC4QPPS3WXESGRE5OVFYPRQT2IUMLDGUBXABC7AF6RN77EOBTA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.175577362 (513h27m34.046660397s) [noise=true , address=145.79.227.15:4874 , piece_id=ZW2UI253CRROCRLLJ45ZV7RCKO63CQN7NWRUGNZNBTGFSPTP5FTA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.176005157 (513h27m34.047088192s) [noise=true , address=145.79.227.38:4978 , piece_id=NF5X2SO4TEC5R274DB5F7PG27SKKDEGB2TKWKZHC33BVKWIXORNA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.176406391 (513h27m34.047489426s) [noise=true , address=145.79.227.6:4258 , piece_id=NDVV4X2IED52XMSIIHKJ3IQ25O5CCQZEDNRZANP6PFE773FUIGOA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.176807234 (513h27m34.047890269s) [noise=true , address=145.79.227.45:4229 , piece_id=HLBM7GOBJDVEZHY5XDFIOMUR6QIOZKBIEJGJXXDLATTEND5GHHBA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.177285144 (513h27m34.048368179s) [noise=true , address=145.79.227.36:4518 , piece_id=G5L5OEAB4Y63BWV3ELJSQI47FO64THL7KLXFSK2IG567KUQQGJLQ, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.178612612 (513h27m34.049695657s) [noise=true , address=145.79.227.19:4079 , piece_id=BPQTBALJDQNUAQLD4FFDP7K2LSUKWY74EUWXE3ZHXJUXCP3N324A, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.178850244 (513h27m34.04993328s) [noise=true , address=145.79.227.39:4014 , piece_id=UIH5AXUO7YAYWJC36ODWZ6CMEWH5UGEXEX6Z4DVRGZ67LABL44FA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.180281249 (513h27m34.051364294s) [noise=true , address=77.90.31.17:4770 , piece_id=Q46SB7DCMSUGTVTSGHF7CKRT4RHWKRCFLROIWVD4UVT7M7VEN5QA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.180363926 (513h27m34.051446971s) [noise=true , address=145.79.227.13:4803 , piece_id=5S75TV3QHKLRNATBPXUEG6JUXCG6UFFORLDI3W42UFKXQ5GZXK6A, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.181493117 (513h27m34.052576152s) [noise=true , address=145.79.227.47:4265 , piece_id=5JGM746GEVIL2SFESK7LBOH4MXII66PVF6PGAGA44U6QQNSHZFZA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.182207617 (513h27m34.053290652s) [noise=true , address=145.79.227.52:4447 , piece_id=ZAY6RYX6UNVEUW5WWZEWRJ2SH52QVQIBTPV2XAU64PKRU6XFLHHQ, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:34.182894656 (513h27m34.053977701s) [noise=true , address=145.79.227.48:4301 , piece_id=UHIQBGU25ZCTEZQRIDSC243A52AJIOBC2EE4UUTVOAJHX5MFSFKA, seg_pos={0 0}, upload=450558]: Uploading piece...
Dec 27 00:56:36.034951419 (513h27m35.906034454s) [upload=450558]: Next returned nil segment
Dec 27 00:56:36.034969614 (513h27m35.906052649s) [upload=450558]: Waiting for error group managing segments...
Dec 27 00:56:36.347990616 (513h27m36.219073651s) [noise=true , address=145.79.227.5:4221 , piece_id=OACPNPTF42HLJ3IZZQ7V2MJX26SXV6J73WAGANUN4HPHVMFWXUYQ, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.355662444 (513h27m36.226745479s) [noise=true , address=145.79.227.45:4229 , piece_id=HLBM7GOBJDVEZHY5XDFIOMUR6QIOZKBIEJGJXXDLATTEND5GHHBA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.358096479 (513h27m36.229179524s) [noise=true , address=145.79.227.15:4874 , piece_id=ZW2UI253CRROCRLLJ45ZV7RCKO63CQN7NWRUGNZNBTGFSPTP5FTA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.359977050 (513h27m36.231060085s) [noise=true , address=145.79.227.52:4447 , piece_id=ZAY6RYX6UNVEUW5WWZEWRJ2SH52QVQIBTPV2XAU64PKRU6XFLHHQ, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.373113073 (513h27m36.244196118s) [noise=true , address=145.79.227.19:4079 , piece_id=BPQTBALJDQNUAQLD4FFDP7K2LSUKWY74EUWXE3ZHXJUXCP3N324A, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.387065871 (513h27m36.258148906s) [noise=true , address=145.79.227.36:4518 , piece_id=G5L5OEAB4Y63BWV3ELJSQI47FO64THL7KLXFSK2IG567KUQQGJLQ, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.387914778 (513h27m36.258997823s) [noise=true , address=145.79.227.6:4258 , piece_id=NDVV4X2IED52XMSIIHKJ3IQ25O5CCQZEDNRZANP6PFE773FUIGOA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.399672566 (513h27m36.270755601s) [noise=true , address=145.79.227.13:4803 , piece_id=5S75TV3QHKLRNATBPXUEG6JUXCG6UFFORLDI3W42UFKXQ5GZXK6A, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.426139071 (513h27m36.297222116s) [noise=true , address=45.91.116.5:4300 , piece_id=KFU42A23RPS6TZOZDNRT6DPBOCCCTKWZRUD2WC4ZUKS5MMVVYKMA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.494313273 (513h27m36.365396308s) [noise=true , address=77.90.31.17:4770 , piece_id=Q46SB7DCMSUGTVTSGHF7CKRT4RHWKRCFLROIWVD4UVT7M7VEN5QA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.591732612 (513h27m36.462815647s) [noise=true , address=45.91.116.12:4225 , piece_id=NHDIPNWZ2SX3C5ERYR47Q3KDCSAAEA55VAPIXBAJGBMSKYYB5VNQ, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:36.640303357 (513h27m36.511386402s) [noise=true , address=145.79.227.47:4265 , piece_id=5JGM746GEVIL2SFESK7LBOH4MXII66PVF6PGAGA44U6QQNSHZFZA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:37.070371981 (513h27m36.941455016s) [noise=true , address=145.79.227.38:4978 , piece_id=NF5X2SO4TEC5R274DB5F7PG27SKKDEGB2TKWKZHC33BVKWIXORNA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: <nil>
Dec 27 00:56:37.070403962 (513h27m36.941487007s) [seg_pos={0 0}, upload=450558]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:56:37.070624612 (513h27m36.941707647s) [noise=true , address=145.79.227.39:4014 , piece_id=UIH5AXUO7YAYWJC36ODWZ6CMEWH5UGEXEX6Z4DVRGZ67LABL44FA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
Dec 27 00:56:37.070631735 (513h27m36.94171477s) [noise=true , address=145.79.227.39:4014 , piece_id=UIH5AXUO7YAYWJC36ODWZ6CMEWH5UGEXEX6Z4DVRGZ67LABL44FA, seg_pos={0 0}, upload=450558]: Long tail cancellation due to optimal threshold
Dec 27 00:56:37.070870079 (513h27m36.941953114s) [noise=true , address=145.79.227.29:4941 , piece_id=QBVC4QPPS3WXESGRE5OVFYPRQT2IUMLDGUBXABC7AF6RN77EOBTA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 27 00:56:37.070877393 (513h27m36.941960428s) [noise=true , address=145.79.227.29:4941 , piece_id=QBVC4QPPS3WXESGRE5OVFYPRQT2IUMLDGUBXABC7AF6RN77EOBTA, seg_pos={0 0}, upload=450558]: Long tail cancellation due to optimal threshold
Dec 27 00:56:37.071004305 (513h27m36.94208734s) [noise=true , address=145.79.227.48:4301 , piece_id=UHIQBGU25ZCTEZQRIDSC243A52AJIOBC2EE4UUTVOAJHX5MFSFKA, seg_pos={0 0}, upload=450558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF
Dec 27 00:56:37.071011268 (513h27m36.942094313s) [noise=true , address=145.79.227.48:4301 , piece_id=UHIQBGU25ZCTEZQRIDSC243A52AJIOBC2EE4UUTVOAJHX5MFSFKA, seg_pos={0 0}, upload=450558]: Long tail cancellation due to optimal threshold
Dec 27 00:56:37.071029664 (513h27m36.942112699s) [seg_pos={0 0}, upload=450558]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:56:37.071110818 (513h27m36.942193853s) [upload=450558]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:56:37.071116418 (513h27m36.942199463s) [upload=450558]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:56:37.365768463 (513h27m37.236851498s) [upload=450558]: Done uploading segments...
Dec 27 00:56:37.365786918 (513h27m37.236869953s) [upload=450558]: Upload finished. err: <nil>
Duration: 30.002776263s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
39.102223802s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:56:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 39.102251884s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [...X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X.X.XXXX] duration:
30.002618657s trace:
https://jaeger.dev.storj.io/trace/73ac9dcfae1c76bd
FAIL (2025-12-27 00:55:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 3DXBSSQSVOKMATYLBS7Z6EIWW44JMEGIRHLDVGZVN6SVGMAUPGKA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: HPYI7BFE267UDKSETXQ4KFAFQK573HV7HUFMG2XUPWJ22V2EAXLA) context deadline exceeded
======LOGS======
Dec 27 00:55:23.641693131 (513h26m23.512776166s) [upload=450541]: Starting upload
Dec 27 00:55:23.641789114 (513h26m23.512872149s) [upload=450541]: Done starting upload
Dec 27 00:55:23.641929201 (513h26m23.513012266s) [upload=450541]: Uploading segments...
Dec 27 00:55:23.642087151 (513h26m23.513170196s) [upload=450541]: Got next segment. Inline: false
Dec 27 00:55:23.642129231 (513h26m23.513212266s) [upload=450541]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:55:23.642137507 (513h26m23.513220542s) [upload=450541]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:55:23.887798777 (513h26m23.758882112s) [seg_pos={0 0}, upload=450541]: Begin upload segment...
Dec 27 00:55:23.887863220 (513h26m23.758946255s) [seg_pos={0 0}, upload=450541]: Done begin upload segment.
Dec 27 00:55:23.887884040 (513h26m23.758967075s) [noise=true , address=145.79.227.11:4732 , piece_id=NPFMZICDRAPJZVH5K6ACPI7LEMR5LVNUW4IYUMDFR3HIIW7ICJKA, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.887917924 (513h26m23.759000969s) [noise=true , address=145.79.227.15:4874 , piece_id=VYLOYCRR6A4GIQZN6J2MMDBRUY43QHKD7KPJEJJANNLX2SKTOY2Q, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.888449506 (513h26m23.759532541s) [noise=true , address=77.90.31.10:4520 , piece_id=QI453E4GRIXN2GDYJ3KKCXVHNDTXETBLNHZJYFCVEHLE4RVCU2JA, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.888703890 (513h26m23.759786925s) [noise=true , address=145.79.227.45:4231 , piece_id=NNSIA3G6WH7FN4H4IHAVHQ54TSUG5HDWWYSA456SWDUG5Y2232AQ, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.888919972 (513h26m23.760003018s) [noise=true , address=77.90.31.14:4664 , piece_id=3DXBSSQSVOKMATYLBS7Z6EIWW44JMEGIRHLDVGZVN6SVGMAUPGKA, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.889040592 (513h26m23.760123637s) [noise=true , address=145.79.227.36:4518 , piece_id=CEGVQBPK4HFAARFPTGW5XTJXVQ2SISTQDFWY53ZWCIBYSC3ETFBQ, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.889348398 (513h26m23.760431443s) [noise=true , address=145.79.227.3:4150 , piece_id=3EGQZGDH7C5ACJNZB5SL6IZDK7OHZMRSTBALLXCRKMAPGO6QEWIQ, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.889367163 (513h26m23.760450198s) [noise=true , address=145.79.227.38:4978 , piece_id=VF7KTDAGQAVFMPBZWHJSCHZEOCPKC7ZHDO4EI3B5SHPSAEZ64NQA, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.889646455 (513h26m23.7607295s) [noise=true , address=45.91.116.6:4973 , piece_id=ACSQCUQARNBMTEYOIW2TF6JY4ANSDXJYZ5ECH2LAIWAKJDBYMR3Q, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.889928944 (513h26m23.761011979s) [noise=true , address=145.79.227.48:4302 , piece_id=UMSQ56RHFSQQW6B76Z7MAEF55SSEMEGCRAV7EI66JME7BL667NVA, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.890180262 (513h26m23.761263307s) [noise=true , address=145.79.227.13:4804 , piece_id=FZRAO7UQB2JP2ZGR7LHXPXVKAHS2YJX4JZIBQFCXS3YRXU4MN3SA, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.890640959 (513h26m23.761724034s) [noise=true , address=45.91.116.11:4117 , piece_id=AIXWLWAUEOWWDGYRYIS56JGEI4PEDPYVZWDRFJKE3A4CQAGD3LOA, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.890691936 (513h26m23.761774971s) [noise=true , address=45.91.116.12:4225 , piece_id=BVBC6DQVJ5XBKEHF4FZ7O6IXU6RAZ6XVO32GWD5FK5KJOJTR5UIA, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.891027305 (513h26m23.76211034s) [noise=true , address=145.79.227.12:4765 , piece_id=APMJ3K7KHPJ42X6KWF32LCHMAWHUQ3IXALZLRR572FEKWBBZY67A, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.891335482 (513h26m23.762418517s) [noise=true , address=145.79.227.49:4339 , piece_id=DRNQVDCRNHI2MCJH6GQV46FMSSOGMOM3Y2MZWDQELUJEM3R2PT7Q, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:23.891405595 (513h26m23.76248863s) [noise=true , address=45.91.116.10:4081 , piece_id=HPYI7BFE267UDKSETXQ4KFAFQK573HV7HUFMG2XUPWJ22V2EAXLA, seg_pos={0 0}, upload=450541]: Uploading piece...
Dec 27 00:55:25.500897295 (513h26m25.37198033s) [upload=450541]: Next returned nil segment
Dec 27 00:55:25.500914347 (513h26m25.371997382s) [upload=450541]: Waiting for error group managing segments...
Dec 27 00:55:25.884077439 (513h26m25.755160474s) [noise=true , address=145.79.227.11:4732 , piece_id=NPFMZICDRAPJZVH5K6ACPI7LEMR5LVNUW4IYUMDFR3HIIW7ICJKA, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:25.891176456 (513h26m25.762259491s) [noise=true , address=145.79.227.12:4765 , piece_id=APMJ3K7KHPJ42X6KWF32LCHMAWHUQ3IXALZLRR572FEKWBBZY67A, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:25.892163927 (513h26m25.763247002s) [noise=true , address=145.79.227.48:4302 , piece_id=UMSQ56RHFSQQW6B76Z7MAEF55SSEMEGCRAV7EI66JME7BL667NVA, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:25.893964535 (513h26m25.76504757s) [noise=true , address=145.79.227.15:4874 , piece_id=VYLOYCRR6A4GIQZN6J2MMDBRUY43QHKD7KPJEJJANNLX2SKTOY2Q, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:25.894076429 (513h26m25.765159464s) [noise=true , address=145.79.227.49:4339 , piece_id=DRNQVDCRNHI2MCJH6GQV46FMSSOGMOM3Y2MZWDQELUJEM3R2PT7Q, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:25.899465921 (513h26m25.770548956s) [noise=true , address=145.79.227.45:4231 , piece_id=NNSIA3G6WH7FN4H4IHAVHQ54TSUG5HDWWYSA456SWDUG5Y2232AQ, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:25.908083119 (513h26m25.779166154s) [noise=true , address=145.79.227.13:4804 , piece_id=FZRAO7UQB2JP2ZGR7LHXPXVKAHS2YJX4JZIBQFCXS3YRXU4MN3SA, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:25.921210055 (513h26m25.79229309s) [noise=true , address=145.79.227.38:4978 , piece_id=VF7KTDAGQAVFMPBZWHJSCHZEOCPKC7ZHDO4EI3B5SHPSAEZ64NQA, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:25.927961069 (513h26m25.799044104s) [noise=true , address=145.79.227.36:4518 , piece_id=CEGVQBPK4HFAARFPTGW5XTJXVQ2SISTQDFWY53ZWCIBYSC3ETFBQ, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:26.133700865 (513h26m26.00478391s) [noise=true , address=45.91.116.12:4225 , piece_id=BVBC6DQVJ5XBKEHF4FZ7O6IXU6RAZ6XVO32GWD5FK5KJOJTR5UIA, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:26.141080265 (513h26m26.01216332s) [noise=true , address=45.91.116.10:4081 , piece_id=HPYI7BFE267UDKSETXQ4KFAFQK573HV7HUFMG2XUPWJ22V2EAXLA, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:26.146005424 (513h26m26.017088459s) [noise=true , address=45.91.116.6:4973 , piece_id=ACSQCUQARNBMTEYOIW2TF6JY4ANSDXJYZ5ECH2LAIWAKJDBYMR3Q, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:26.154681053 (513h26m26.025764088s) [noise=true , address=77.90.31.14:4664 , piece_id=3DXBSSQSVOKMATYLBS7Z6EIWW44JMEGIRHLDVGZVN6SVGMAUPGKA, seg_pos={0 0}, upload=450541]: Done uploading piece. err: <nil>
Dec 27 00:55:26.154707784 (513h26m26.025790829s) [seg_pos={0 0}, upload=450541]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:55:26.154929316 (513h26m26.026012372s) [noise=true , address=145.79.227.3:4150 , piece_id=3EGQZGDH7C5ACJNZB5SL6IZDK7OHZMRSTBALLXCRKMAPGO6QEWIQ, seg_pos={0 0}, upload=450541]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:33144->145.79.227.3:4150: use of closed network connection; write tcp 188.166.225.170:33144->145.79.227.3:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33144->145.79.227.3:4150: use of closed network connection
Dec 27 00:55:26.154937432 (513h26m26.026020477s) [noise=true , address=145.79.227.3:4150 , piece_id=3EGQZGDH7C5ACJNZB5SL6IZDK7OHZMRSTBALLXCRKMAPGO6QEWIQ, seg_pos={0 0}, upload=450541]: Long tail cancellation due to optimal threshold
Dec 27 00:55:26.154951077 (513h26m26.026034112s) [noise=true , address=45.91.116.11:4117 , piece_id=AIXWLWAUEOWWDGYRYIS56JGEI4PEDPYVZWDRFJKE3A4CQAGD3LOA, seg_pos={0 0}, upload=450541]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 27 00:55:26.154960896 (513h26m26.026043931s) [noise=true , address=45.91.116.11:4117 , piece_id=AIXWLWAUEOWWDGYRYIS56JGEI4PEDPYVZWDRFJKE3A4CQAGD3LOA, seg_pos={0 0}, upload=450541]: Long tail cancellation due to optimal threshold
Dec 27 00:55:26.155100992 (513h26m26.026184028s) [noise=true , address=77.90.31.10:4520 , piece_id=QI453E4GRIXN2GDYJ3KKCXVHNDTXETBLNHZJYFCVEHLE4RVCU2JA, seg_pos={0 0}, upload=450541]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 27 00:55:26.155108147 (513h26m26.026191182s) [noise=true , address=77.90.31.10:4520 , piece_id=QI453E4GRIXN2GDYJ3KKCXVHNDTXETBLNHZJYFCVEHLE4RVCU2JA, seg_pos={0 0}, upload=450541]: Long tail cancellation due to optimal threshold
Dec 27 00:55:26.155196434 (513h26m26.02627948s) [seg_pos={0 0}, upload=450541]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:55:26.155277880 (513h26m26.026360915s) [upload=450541]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:55:26.155283020 (513h26m26.026366055s) [upload=450541]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:55:26.440892697 (513h26m26.311975762s) [upload=450541]: Done uploading segments...
Dec 27 00:55:26.440917104 (513h26m26.312000149s) [upload=450541]: Upload finished. err: <nil>
Duration: 30.002639065s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
27.62941963s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:55:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 27.629438415s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [....X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X.X.XXX] duration:
30.004580198s trace:
https://jaeger.dev.storj.io/trace/380573d2e99fce48
FAIL (2025-12-27 00:55:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: YEIQEYEOCBPNPSLWHHLOYLQDBYS6MSNQOUQGUQM275RRINVQWLRA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 7JL7DERD4ZRSLO7C6NO6H26M6BXDTWBJWLPF6KMMBCBRSOXCD3PA) context deadline exceeded
======LOGS======
9533 (513h25m35.850232578s) [upload=450527]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:54:36.224790916 (513h25m36.095873971s) [seg_pos={0 0}, upload=450527]: Begin upload segment...
Dec 27 00:54:36.224848906 (513h25m36.095931951s) [seg_pos={0 0}, upload=450527]: Done begin upload segment.
Dec 27 00:54:36.224865237 (513h25m36.095948282s) [noise=true , address=145.79.227.48:4303 , piece_id=3HAXTCSDXQJZVWA3GN3BYDGLDVY5LPNJYT2APIRX67RDS6CX3N7A, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.224936723 (513h25m36.096019768s) [noise=true , address=145.79.227.17:4972 , piece_id=3IKY2DYB3FUDJXRPRTDVBFSVV4HNDMTYL5BDSQXBT73DCICYPLWA, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.225418280 (513h25m36.096501315s) [noise=true , address=145.79.227.5:4223 , piece_id=HB77OT5U7ZV43U3P4WZW6L2RHUKVJWNHUFD4OGX22IVBGRFLLM3Q, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.225904095 (513h25m36.09698714s) [noise=true , address=45.91.116.7:4011 , piece_id=7JL7DERD4ZRSLO7C6NO6H26M6BXDTWBJWLPF6KMMBCBRSOXCD3PA, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.226602395 (513h25m36.09768544s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIJB57FCXNK2DYDNLF2HQY4IWLOF6MUW3EKCXCTNQ5YDNR5WH3MA, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.227006514 (513h25m36.098089549s) [noise=true , address=45.91.116.4:4940 , piece_id=BL353KANEJ6YLBKRK5CVGSVSVCTFLRNDBXCNVQMUG47FNUFH72GA, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.227500355 (513h25m36.09858339s) [noise=true , address=145.79.227.13:4803 , piece_id=E4LN4I7BR5OYHB54WTEB2P2OCJBTMKET23VBKQPB6FIIA2MJPHEA, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.229897149 (513h25m36.100980194s) [noise=true , address=145.79.227.2:4115 , piece_id=KJZV432JSGKBC2Z7SKNBLMUMIJ5SJIA43T5HZPWBDUGH3THFGFDA, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.231693138 (513h25m36.102776183s) [noise=true , address=145.79.227.11:4732 , piece_id=D6UJE5KCI42EOIFEN25OE37HTQMICC4OAKRIXM7ZNU2VGZQILIJA, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.233203725 (513h25m36.10428677s) [noise=true , address=145.79.227.3:4149 , piece_id=VHCBX22FAUHWXPZR3M5HBNSDID6WKLS6SQOT6FAOVWQHV6FFP3YQ, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.234440279 (513h25m36.105523315s) [noise=true , address=145.79.227.47:4265 , piece_id=H6EGSIIEIOPU4HD3NAYFJWY7DYIBBGHTGRWQUOBNQYMJCX2VPOYA, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.235157955 (513h25m36.106240991s) [noise=true , address=45.91.116.11:4119 , piece_id=RIWTN4SE52RJ2YIEUWMXZR7UYWAOJXWDC3UXO36XY5I7VQIED36A, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.236181505 (513h25m36.10726454s) [noise=true , address=77.90.31.16:4736 , piece_id=YEIQEYEOCBPNPSLWHHLOYLQDBYS6MSNQOUQGUQM275RRINVQWLRA, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.236533135 (513h25m36.10761617s) [noise=true , address=45.91.116.10:4081 , piece_id=7Q6GVRNV6ZTOF7YK7WSV5ZN5O62YBWHC257UFSWOYC7MVHQHYF3A, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.237714093 (513h25m36.108797138s) [noise=true , address=145.79.227.42:4121 , piece_id=JJDBHG2ZOLDYDCRMK5VBVX5PU56ZOSSLGRCYNRU4VF26ZV7JLH6Q, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:36.239020951 (513h25m36.110103986s) [noise=true , address=145.79.227.51:4411 , piece_id=6U3WGV5IKSI4VDPLICIUQDXSBXIMWC6YNK75RQKKYS4KPAFYN7NQ, seg_pos={0 0}, upload=450527]: Uploading piece...
Dec 27 00:54:37.947391961 (513h25m37.818474996s) [upload=450527]: Next returned nil segment
Dec 27 00:54:37.947404053 (513h25m37.818487089s) [upload=450527]: Waiting for error group managing segments...
Dec 27 00:54:38.222314027 (513h25m38.093397072s) [noise=true , address=145.79.227.11:4732 , piece_id=D6UJE5KCI42EOIFEN25OE37HTQMICC4OAKRIXM7ZNU2VGZQILIJA, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.238386973 (513h25m38.109470008s) [noise=true , address=145.79.227.42:4121 , piece_id=JJDBHG2ZOLDYDCRMK5VBVX5PU56ZOSSLGRCYNRU4VF26ZV7JLH6Q, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.247277842 (513h25m38.118360877s) [noise=true , address=145.79.227.48:4303 , piece_id=3HAXTCSDXQJZVWA3GN3BYDGLDVY5LPNJYT2APIRX67RDS6CX3N7A, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.250600950 (513h25m38.121683985s) [noise=true , address=145.79.227.3:4149 , piece_id=VHCBX22FAUHWXPZR3M5HBNSDID6WKLS6SQOT6FAOVWQHV6FFP3YQ, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.256178330 (513h25m38.127261365s) [noise=true , address=145.79.227.17:4972 , piece_id=3IKY2DYB3FUDJXRPRTDVBFSVV4HNDMTYL5BDSQXBT73DCICYPLWA, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.260071202 (513h25m38.131154237s) [noise=true , address=145.79.227.2:4115 , piece_id=KJZV432JSGKBC2Z7SKNBLMUMIJ5SJIA43T5HZPWBDUGH3THFGFDA, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.263857943 (513h25m38.134940988s) [noise=true , address=145.79.227.13:4803 , piece_id=E4LN4I7BR5OYHB54WTEB2P2OCJBTMKET23VBKQPB6FIIA2MJPHEA, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.276180997 (513h25m38.147264042s) [noise=true , address=145.79.227.47:4265 , piece_id=H6EGSIIEIOPU4HD3NAYFJWY7DYIBBGHTGRWQUOBNQYMJCX2VPOYA, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.487569208 (513h25m38.358652243s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIJB57FCXNK2DYDNLF2HQY4IWLOF6MUW3EKCXCTNQ5YDNR5WH3MA, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.522800062 (513h25m38.393883097s) [noise=true , address=45.91.116.7:4011 , piece_id=7JL7DERD4ZRSLO7C6NO6H26M6BXDTWBJWLPF6KMMBCBRSOXCD3PA, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.546958613 (513h25m38.418041648s) [noise=true , address=77.90.31.16:4736 , piece_id=YEIQEYEOCBPNPSLWHHLOYLQDBYS6MSNQOUQGUQM275RRINVQWLRA, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.558431860 (513h25m38.429514895s) [noise=true , address=45.91.116.11:4119 , piece_id=RIWTN4SE52RJ2YIEUWMXZR7UYWAOJXWDC3UXO36XY5I7VQIED36A, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.577897465 (513h25m38.4489805s) [noise=true , address=45.91.116.4:4940 , piece_id=BL353KANEJ6YLBKRK5CVGSVSVCTFLRNDBXCNVQMUG47FNUFH72GA, seg_pos={0 0}, upload=450527]: Done uploading piece. err: <nil>
Dec 27 00:54:38.577915418 (513h25m38.448998464s) [seg_pos={0 0}, upload=450527]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:54:38.578077527 (513h25m38.449160563s) [noise=true , address=145.79.227.5:4223 , piece_id=HB77OT5U7ZV43U3P4WZW6L2RHUKVJWNHUFD4OGX22IVBGRFLLM3Q, seg_pos={0 0}, upload=450527]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:39296->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:39296->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39296->145.79.227.5:4223: use of closed network connection
Dec 27 00:54:38.578083469 (513h25m38.449166514s) [noise=true , address=145.79.227.5:4223 , piece_id=HB77OT5U7ZV43U3P4WZW6L2RHUKVJWNHUFD4OGX22IVBGRFLLM3Q, seg_pos={0 0}, upload=450527]: Long tail cancellation due to optimal threshold
Dec 27 00:54:38.578299339 (513h25m38.449382375s) [noise=true , address=45.91.116.10:4081 , piece_id=7Q6GVRNV6ZTOF7YK7WSV5ZN5O62YBWHC257UFSWOYC7MVHQHYF3A, seg_pos={0 0}, upload=450527]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:53194->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:53194->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53194->45.91.116.10:4081: use of closed network connection
Dec 27 00:54:38.578308367 (513h25m38.449391402s) [noise=true , address=45.91.116.10:4081 , piece_id=7Q6GVRNV6ZTOF7YK7WSV5ZN5O62YBWHC257UFSWOYC7MVHQHYF3A, seg_pos={0 0}, upload=450527]: Long tail cancellation due to optimal threshold
Dec 27 00:54:38.578315080 (513h25m38.449398115s) [noise=true , address=145.79.227.51:4411 , piece_id=6U3WGV5IKSI4VDPLICIUQDXSBXIMWC6YNK75RQKKYS4KPAFYN7NQ, seg_pos={0 0}, upload=450527]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:59710->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:59710->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59710->145.79.227.51:4411: use of closed network connection
Dec 27 00:54:38.578320951 (513h25m38.449403986s) [noise=true , address=145.79.227.51:4411 , piece_id=6U3WGV5IKSI4VDPLICIUQDXSBXIMWC6YNK75RQKKYS4KPAFYN7NQ, seg_pos={0 0}, upload=450527]: Long tail cancellation due to optimal threshold
Dec 27 00:54:38.578427885 (513h25m38.44951092s) [seg_pos={0 0}, upload=450527]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:54:38.578495994 (513h25m38.449579029s) [upload=450527]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:54:38.578499912 (513h25m38.449582947s) [upload=450527]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:54:38.873274110 (513h25m38.744357145s) [upload=450527]: Done uploading segments...
Dec 27 00:54:38.873293146 (513h25m38.744376181s) [upload=450527]: Upload finished. err: <nil>
Duration: 30.004605939s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
8.481727452s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:54:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 8.48175801s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
18.850536959s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:54:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 18.850565022s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [.....X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X.X.XX] duration:
30.003201388s trace:
https://jaeger.dev.storj.io/trace/649d44ac9271007d
FAIL (2025-12-27 00:54:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: XFYTSRKA7AO2CY2NWBYDGMULBVGN62NZGSWANR4Y7K3GUEPHTE3A) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KKFMPJJH27AZ3YYA72V3DXHVC2SNWO2HJX5WUQZNNVIY3Z7M7ZPQ) context deadline exceeded
======LOGS======
Dec 27 00:53:35.093391298 (513h24m34.964474343s) [upload=450511]: Starting upload
Dec 27 00:53:35.093547796 (513h24m34.964630831s) [upload=450511]: Done starting upload
Dec 27 00:53:35.093751124 (513h24m34.964834159s) [upload=450511]: Uploading segments...
Dec 27 00:53:35.093788385 (513h24m34.96487142s) [upload=450511]: Got next segment. Inline: false
Dec 27 00:53:35.093832248 (513h24m34.964915283s) [upload=450511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:53:35.093836806 (513h24m34.964919841s) [upload=450511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:53:35.345431305 (513h24m35.21651435s) [seg_pos={0 0}, upload=450511]: Begin upload segment...
Dec 27 00:53:35.345485509 (513h24m35.216568554s) [seg_pos={0 0}, upload=450511]: Done begin upload segment.
Dec 27 00:53:35.345540954 (513h24m35.216623999s) [noise=true , address=45.91.116.8:4046 , piece_id=XFYTSRKA7AO2CY2NWBYDGMULBVGN62NZGSWANR4Y7K3GUEPHTE3A, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.345562415 (513h24m35.21664548s) [noise=true , address=77.90.31.18:4808 , piece_id=KKFMPJJH27AZ3YYA72V3DXHVC2SNWO2HJX5WUQZNNVIY3Z7M7ZPQ, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.346380593 (513h24m35.217463628s) [noise=true , address=145.79.227.36:4517 , piece_id=DGSHAJHKVEJTMNEQ2I7C4DHCGJFXPV4GPFTI43CBP3SPDCGVQ2GA, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.346580423 (513h24m35.217663478s) [noise=true , address=77.90.31.21:4877 , piece_id=DTGEXUP5GMHAUO56SIZBQAALRUIFNOT6DM4NHISJUELZHC5RBG3A, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.346938435 (513h24m35.21802148s) [noise=true , address=145.79.227.50:4375 , piece_id=R7P3YEVL5SOLPTURRAJU5EXI3LCGOFBDDUWUJOC5ZBCSUXZ5PWFA, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.347037043 (513h24m35.218120088s) [noise=true , address=145.79.227.45:4229 , piece_id=7BK37A4GJ7J6AITEEX2D2OKW2L5VMLNPNTDHFCL2UGPUV5DRWQAQ, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.347420413 (513h24m35.218503468s) [noise=true , address=145.79.227.52:4445 , piece_id=NTDV67CH5VSKONI2VM5TXM7F5ENSCSG7IW5RYINAGUXTFXWOMKTQ, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.347458956 (513h24m35.218541991s) [noise=true , address=145.79.227.49:4339 , piece_id=VFTKZMGXR3UXTXOMG2RKXW4ILD5HHQFHHLKXYFKKF6T3STL25RIA, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.347832958 (513h24m35.218915993s) [noise=true , address=145.79.227.33:4869 , piece_id=3LFHUI6SPZKM7PYBVB3HZBEJ2UA6QEJH6RKDWUQNQCQNT2N5QYYA, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.347906328 (513h24m35.218989363s) [noise=true , address=145.79.227.51:4409 , piece_id=S4LDJD3Y66VPJVKZ3HTOMWLWHRVZJ3QHOLAUIPDRGUEU7BUOPECA, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.348217240 (513h24m35.219300285s) [noise=true , address=45.91.116.9:4261 , piece_id=U5WDEHKQTQ3QF7WAYACDAAUUPM2O7QAZSN5FO2R6R5COYMOU26KQ, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.348358479 (513h24m35.219441514s) [noise=true , address=77.90.31.13:4627 , piece_id=XIJUQILWNE4HSCKWA3ZZVC6YSP3PAP5WGH6CLMSLWHTNEII6R2EQ, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.348562267 (513h24m35.219645312s) [noise=true , address=77.90.31.19:4842 , piece_id=S2HWEMLE3NFJQ3PPKLNV5NRZYJN367O7ZXEZF4JKSFILIVSOQOHQ, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.348808847 (513h24m35.219891882s) [noise=true , address=45.91.116.4:4939 , piece_id=NXNHYFSCIEZREPOGTMQ424BQH2Z2475ZOR5FAVN5EJBSDYN63Z3A, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.348905240 (513h24m35.219988275s) [noise=true , address=45.91.116.5:4300 , piece_id=YMY7JN7BUWL7VGUJTZZLCL2IMJYQKD5G7XD5INSBIHBL2IBJCSPQ, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:35.349321522 (513h24m35.220404557s) [noise=true , address=145.79.227.11:4729 , piece_id=JCCAU55IFF2GEJFSFADJM6OXL6JG35ATXARANHVOTGYIENL7OZDA, seg_pos={0 0}, upload=450511]: Uploading piece...
Dec 27 00:53:37.232084446 (513h24m37.103167481s) [upload=450511]: Next returned nil segment
Dec 27 00:53:37.232095607 (513h24m37.103178642s) [upload=450511]: Waiting for error group managing segments...
Dec 27 00:53:37.532375599 (513h24m37.403458634s) [noise=true , address=77.90.31.19:4842 , piece_id=S2HWEMLE3NFJQ3PPKLNV5NRZYJN367O7ZXEZF4JKSFILIVSOQOHQ, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:37.558337915 (513h24m37.42942097s) [noise=true , address=77.90.31.13:4627 , piece_id=XIJUQILWNE4HSCKWA3ZZVC6YSP3PAP5WGH6CLMSLWHTNEII6R2EQ, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:37.561025803 (513h24m37.432108848s) [noise=true , address=145.79.227.51:4409 , piece_id=S4LDJD3Y66VPJVKZ3HTOMWLWHRVZJ3QHOLAUIPDRGUEU7BUOPECA, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:37.590389214 (513h24m37.461472249s) [noise=true , address=45.91.116.9:4261 , piece_id=U5WDEHKQTQ3QF7WAYACDAAUUPM2O7QAZSN5FO2R6R5COYMOU26KQ, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:37.597352724 (513h24m37.468435769s) [noise=true , address=77.90.31.21:4877 , piece_id=DTGEXUP5GMHAUO56SIZBQAALRUIFNOT6DM4NHISJUELZHC5RBG3A, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:37.605447628 (513h24m37.476530663s) [noise=true , address=145.79.227.11:4729 , piece_id=JCCAU55IFF2GEJFSFADJM6OXL6JG35ATXARANHVOTGYIENL7OZDA, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:37.617128850 (513h24m37.488211895s) [noise=true , address=45.91.116.5:4300 , piece_id=YMY7JN7BUWL7VGUJTZZLCL2IMJYQKD5G7XD5INSBIHBL2IBJCSPQ, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:37.631878435 (513h24m37.50296149s) [noise=true , address=45.91.116.8:4046 , piece_id=XFYTSRKA7AO2CY2NWBYDGMULBVGN62NZGSWANR4Y7K3GUEPHTE3A, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:37.634407361 (513h24m37.505490406s) [noise=true , address=45.91.116.4:4939 , piece_id=NXNHYFSCIEZREPOGTMQ424BQH2Z2475ZOR5FAVN5EJBSDYN63Z3A, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:37.862010847 (513h24m37.733093882s) [noise=true , address=145.79.227.36:4517 , piece_id=DGSHAJHKVEJTMNEQ2I7C4DHCGJFXPV4GPFTI43CBP3SPDCGVQ2GA, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:38.245891205 (513h24m38.11697425s) [noise=true , address=77.90.31.18:4808 , piece_id=KKFMPJJH27AZ3YYA72V3DXHVC2SNWO2HJX5WUQZNNVIY3Z7M7ZPQ, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:38.264343772 (513h24m38.135426807s) [noise=true , address=145.79.227.45:4229 , piece_id=7BK37A4GJ7J6AITEEX2D2OKW2L5VMLNPNTDHFCL2UGPUV5DRWQAQ, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:38.517284081 (513h24m38.388367116s) [noise=true , address=145.79.227.52:4445 , piece_id=NTDV67CH5VSKONI2VM5TXM7F5ENSCSG7IW5RYINAGUXTFXWOMKTQ, seg_pos={0 0}, upload=450511]: Done uploading piece. err: <nil>
Dec 27 00:53:38.517299791 (513h24m38.388382826s) [seg_pos={0 0}, upload=450511]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:53:38.517509380 (513h24m38.388592425s) [noise=true , address=145.79.227.33:4869 , piece_id=3LFHUI6SPZKM7PYBVB3HZBEJ2UA6QEJH6RKDWUQNQCQNT2N5QYYA, seg_pos={0 0}, upload=450511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:53:38.517516914 (513h24m38.388599949s) [noise=true , address=145.79.227.33:4869 , piece_id=3LFHUI6SPZKM7PYBVB3HZBEJ2UA6QEJH6RKDWUQNQCQNT2N5QYYA, seg_pos={0 0}, upload=450511]: Long tail cancellation due to optimal threshold
Dec 27 00:53:38.517731553 (513h24m38.388814588s) [noise=true , address=145.79.227.50:4375 , piece_id=R7P3YEVL5SOLPTURRAJU5EXI3LCGOFBDDUWUJOC5ZBCSUXZ5PWFA, seg_pos={0 0}, upload=450511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF
Dec 27 00:53:38.517737364 (513h24m38.388820399s) [noise=true , address=145.79.227.50:4375 , piece_id=R7P3YEVL5SOLPTURRAJU5EXI3LCGOFBDDUWUJOC5ZBCSUXZ5PWFA, seg_pos={0 0}, upload=450511]: Long tail cancellation due to optimal threshold
Dec 27 00:53:38.518100275 (513h24m38.38918332s) [noise=true , address=145.79.227.49:4339 , piece_id=VFTKZMGXR3UXTXOMG2RKXW4ILD5HHQFHHLKXYFKKF6T3STL25RIA, seg_pos={0 0}, upload=450511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 27 00:53:38.518116526 (513h24m38.389199571s) [noise=true , address=145.79.227.49:4339 , piece_id=VFTKZMGXR3UXTXOMG2RKXW4ILD5HHQFHHLKXYFKKF6T3STL25RIA, seg_pos={0 0}, upload=450511]: Long tail cancellation due to optimal threshold
Dec 27 00:53:38.518143538 (513h24m38.389226573s) [seg_pos={0 0}, upload=450511]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:53:38.518272373 (513h24m38.389355408s) [upload=450511]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:53:38.518278404 (513h24m38.38936146s) [upload=450511]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:53:38.809826980 (513h24m38.680910015s) [upload=450511]: Done uploading segments...
Dec 27 00:53:38.809846748 (513h24m38.680929783s) [upload=450511]: Upload finished. err: <nil>
Duration: 30.003218981s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
8.266134868s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:54:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 8.266166127s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
17.040706294s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:53:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 17.040721654s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X.X.X] duration:
30.003368152s trace:
https://jaeger.dev.storj.io/trace/6f46ac7e6f25034f
FAIL (2025-12-27 00:53:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F, Piece ID: 34VCHBAK7CKECVUW6CN2HSD3W3DV32EFVJKDC3ECML5D7BOKQHGA) context deadline exceeded; piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: LLNYRROZC4MC2572CVXQBUM2X5FPMGUTTRCEWON2G5XKZF5ZZURQ) context deadline exceeded
======LOGS======
d=450502]: Got next segment. Inline: false
Dec 27 00:53:03.035994139 (513h24m2.907077184s) [upload=450502]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:53:03.035999870 (513h24m2.907082915s) [upload=450502]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:53:03.281295615 (513h24m3.15237866s) [seg_pos={0 0}, upload=450502]: Begin upload segment...
Dec 27 00:53:03.281358936 (513h24m3.152441981s) [seg_pos={0 0}, upload=450502]: Done begin upload segment.
Dec 27 00:53:03.281406065 (513h24m3.15248914s) [noise=true , address=77.90.31.19:4841 , piece_id=LWHAOQRZDBVDE6BHCFY2IFWDD7FP6IRKLJPJHKNWNTXMXGCWP4UQ, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.281546773 (513h24m3.152629828s) [noise=true , address=145.79.227.37:4566 , piece_id=34VCHBAK7CKECVUW6CN2HSD3W3DV32EFVJKDC3ECML5D7BOKQHGA, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.282315236 (513h24m3.153398281s) [noise=true , address=145.79.227.49:4339 , piece_id=LLNYRROZC4MC2572CVXQBUM2X5FPMGUTTRCEWON2G5XKZF5ZZURQ, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.282819908 (513h24m3.153902953s) [noise=true , address=77.90.31.15:4700 , piece_id=6AWR7W6RGVRJ3HWQSLAWR66SB6FCAOR77D5L2SRMFSV2XVYNQLIA, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.283618287 (513h24m3.154701332s) [noise=true , address=45.91.116.5:4298 , piece_id=5BBVYVEEWJ666MWEGUB4ZVQSKPILQGEFJ7VULRZATKOY3RZET7QQ, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.284201598 (513h24m3.155284643s) [noise=true , address=45.91.116.13:4155 , piece_id=WTJEKQ42WCJLDGBIDJHCNEL76CDNIQGX7PDMW4KSH5V7VWVRNLWQ, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.284793755 (513h24m3.1558768s) [noise=true , address=77.90.31.16:4735 , piece_id=35Y6OXLICD72MNU3QHBJ3EMIGRTJTG53BKJUJEYKVHGDIWG6COPA, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.286282420 (513h24m3.157365465s) [noise=true , address=77.90.31.17:4773 , piece_id=MGOZGLDRQDD7Z664AXY3OLWYRHWVMCFZFXWUAHB66GBVHLJJNOIQ, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.287239993 (513h24m3.158323038s) [noise=true , address=145.79.227.52:4445 , piece_id=EHWGVKSPOFS6AE2UHTVBKGAHDWBZXF5MH4TNJAK65RWVHAJ4I2XQ, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.288223355 (513h24m3.1593064s) [noise=true , address=145.79.227.39:4015 , piece_id=OLLGCPP6DYLJQPWUSE2AVWVTWF5CNGRPYUUXYJT5BY5FKVD27IKA, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.289621468 (513h24m3.160704504s) [noise=true , address=145.79.227.15:4874 , piece_id=FCOIM2MBKTCOMQZQT2RUJCSEB7QK3V3P43GQR6KGBOAWBPRI47BA, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.290114397 (513h24m3.161197452s) [noise=true , address=45.91.116.8:4046 , piece_id=B47JBCEY7UZASNE3J6WJ7AEIVQAQSDTCIMUU77V6T373ZBMJ2XWQ, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.290892578 (513h24m3.161975633s) [noise=true , address=145.79.227.50:4373 , piece_id=TWAAHWMNDBZEGRDQEMPKIDD2VVFEWLKPZBRANCSPPWGDWKYJS6ZA, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.292296712 (513h24m3.163379747s) [noise=true , address=145.79.227.46:4602 , piece_id=HDWEXFM7DBBHYE4SN2TZC334WU65ZN7B6WBLKJ7MHDN4ZT57QYOQ, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.293971392 (513h24m3.165054437s) [noise=true , address=145.79.227.38:4977 , piece_id=HQGJ3EGJIORY7MQ5F7JXQPZRAWS2VTBLIJVVLVINZLVMBFVNKFAA, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:03.294976364 (513h24m3.166059409s) [noise=true , address=45.91.116.11:4119 , piece_id=57WKZ2MW6UXNEX2HPFS47LDL46FFUKCYBD7MDHG5HVAJ3KSXFIKA, seg_pos={0 0}, upload=450502]: Uploading piece...
Dec 27 00:53:05.126801034 (513h24m4.99788408s) [upload=450502]: Next returned nil segment
Dec 27 00:53:05.126818367 (513h24m4.997901412s) [upload=450502]: Waiting for error group managing segments...
Dec 27 00:53:05.474960725 (513h24m5.34604376s) [noise=true , address=145.79.227.15:4874 , piece_id=FCOIM2MBKTCOMQZQT2RUJCSEB7QK3V3P43GQR6KGBOAWBPRI47BA, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.480341010 (513h24m5.351424045s) [noise=true , address=145.79.227.37:4566 , piece_id=34VCHBAK7CKECVUW6CN2HSD3W3DV32EFVJKDC3ECML5D7BOKQHGA, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.483377431 (513h24m5.354460466s) [noise=true , address=145.79.227.49:4339 , piece_id=LLNYRROZC4MC2572CVXQBUM2X5FPMGUTTRCEWON2G5XKZF5ZZURQ, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.497153072 (513h24m5.368236117s) [noise=true , address=45.91.116.13:4155 , piece_id=WTJEKQ42WCJLDGBIDJHCNEL76CDNIQGX7PDMW4KSH5V7VWVRNLWQ, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.498956586 (513h24m5.370039631s) [noise=true , address=77.90.31.17:4773 , piece_id=MGOZGLDRQDD7Z664AXY3OLWYRHWVMCFZFXWUAHB66GBVHLJJNOIQ, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.503311028 (513h24m5.374394073s) [noise=true , address=77.90.31.16:4735 , piece_id=35Y6OXLICD72MNU3QHBJ3EMIGRTJTG53BKJUJEYKVHGDIWG6COPA, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.508385229 (513h24m5.379468274s) [noise=true , address=145.79.227.39:4015 , piece_id=OLLGCPP6DYLJQPWUSE2AVWVTWF5CNGRPYUUXYJT5BY5FKVD27IKA, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.518552589 (513h24m5.389635624s) [noise=true , address=45.91.116.5:4298 , piece_id=5BBVYVEEWJ666MWEGUB4ZVQSKPILQGEFJ7VULRZATKOY3RZET7QQ, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.530007562 (513h24m5.401090607s) [noise=true , address=77.90.31.19:4841 , piece_id=LWHAOQRZDBVDE6BHCFY2IFWDD7FP6IRKLJPJHKNWNTXMXGCWP4UQ, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.535429154 (513h24m5.406512199s) [noise=true , address=145.79.227.52:4445 , piece_id=EHWGVKSPOFS6AE2UHTVBKGAHDWBZXF5MH4TNJAK65RWVHAJ4I2XQ, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.558874648 (513h24m5.429957693s) [noise=true , address=45.91.116.11:4119 , piece_id=57WKZ2MW6UXNEX2HPFS47LDL46FFUKCYBD7MDHG5HVAJ3KSXFIKA, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.597161742 (513h24m5.468244777s) [noise=true , address=77.90.31.15:4700 , piece_id=6AWR7W6RGVRJ3HWQSLAWR66SB6FCAOR77D5L2SRMFSV2XVYNQLIA, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.624068656 (513h24m5.495151691s) [noise=true , address=45.91.116.8:4046 , piece_id=B47JBCEY7UZASNE3J6WJ7AEIVQAQSDTCIMUU77V6T373ZBMJ2XWQ, seg_pos={0 0}, upload=450502]: Done uploading piece. err: <nil>
Dec 27 00:53:05.624094926 (513h24m5.495177961s) [seg_pos={0 0}, upload=450502]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:53:05.624429042 (513h24m5.495512077s) [noise=true , address=145.79.227.38:4977 , piece_id=HQGJ3EGJIORY7MQ5F7JXQPZRAWS2VTBLIJVVLVINZLVMBFVNKFAA, seg_pos={0 0}, upload=450502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:36476->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 00:53:05.624436887 (513h24m5.495519932s) [noise=true , address=145.79.227.38:4977 , piece_id=HQGJ3EGJIORY7MQ5F7JXQPZRAWS2VTBLIJVVLVINZLVMBFVNKFAA, seg_pos={0 0}, upload=450502]: Long tail cancellation due to optimal threshold
Dec 27 00:53:05.624478737 (513h24m5.495561792s) [noise=true , address=145.79.227.46:4602 , piece_id=HDWEXFM7DBBHYE4SN2TZC334WU65ZN7B6WBLKJ7MHDN4ZT57QYOQ, seg_pos={0 0}, upload=450502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: write tcp 188.166.225.170:48198->145.79.227.46:4602: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.46:4602: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.46:4602: use of closed network connection
Dec 27 00:53:05.624489267 (513h24m5.495572312s) [noise=true , address=145.79.227.46:4602 , piece_id=HDWEXFM7DBBHYE4SN2TZC334WU65ZN7B6WBLKJ7MHDN4ZT57QYOQ, seg_pos={0 0}, upload=450502]: Long tail cancellation due to optimal threshold
Dec 27 00:53:05.624540654 (513h24m5.495623689s) [noise=true , address=145.79.227.50:4373 , piece_id=TWAAHWMNDBZEGRDQEMPKIDD2VVFEWLKPZBRANCSPPWGDWKYJS6ZA, seg_pos={0 0}, upload=450502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:52152->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.50:4373: use of closed network connection
Dec 27 00:53:05.624545974 (513h24m5.495629009s) [noise=true , address=145.79.227.50:4373 , piece_id=TWAAHWMNDBZEGRDQEMPKIDD2VVFEWLKPZBRANCSPPWGDWKYJS6ZA, seg_pos={0 0}, upload=450502]: Long tail cancellation due to optimal threshold
Dec 27 00:53:05.624617400 (513h24m5.495700435s) [seg_pos={0 0}, upload=450502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:53:05.624725497 (513h24m5.495808542s) [upload=450502]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:53:05.624742118 (513h24m5.495825153s) [upload=450502]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:53:05.911147912 (513h24m5.782230957s) [upload=450502]: Done uploading segments...
Dec 27 00:53:05.911186064 (513h24m5.782269099s) [upload=450502]: Upload finished. err: <nil>
Duration: 30.003390635s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
52.207373644s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:53:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 52.207400927s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
26 out of 60 is failed (threshold=3) [.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X.X.] duration:
6.56192567s trace:
https://jaeger.dev.storj.io/trace/2f2774bd1397dd6b
FAIL (2025-12-27 00:52:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: JFN2CPAUXOKQHCWS4KFHEOT46CLBYYSQFHTGNEFPOBO7LX54AI3A) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: QW4UBY4XGQGBTUIG26DV5IDPVLVY5P7GFRWFRBORPBXKRBE2EPEA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: XBLDSUF7Y5ZHLYQ3OYSC47QG5PQTJN545XUQDXSWAPFOREF3VC6Q) context deadline exceeded
======LOGS======
Dec 27 00:51:00.561417422 (513h22m0.432500467s) [upload=450474]: Starting upload
Dec 27 00:51:00.561475021 (513h22m0.432558056s) [upload=450474]: Done starting upload
Dec 27 00:51:00.561611852 (513h22m0.432694897s) [upload=450474]: Uploading segments...
Dec 27 00:51:00.561709558 (513h22m0.432792603s) [upload=450474]: Got next segment. Inline: false
Dec 27 00:51:00.561766978 (513h22m0.432850023s) [upload=450474]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:51:00.561773741 (513h22m0.432856786s) [upload=450474]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:51:00.803749764 (513h22m0.6748328s) [seg_pos={0 0}, upload=450474]: Begin upload segment...
Dec 27 00:51:00.803798527 (513h22m0.674881562s) [seg_pos={0 0}, upload=450474]: Done begin upload segment.
Dec 27 00:51:00.803824046 (513h22m0.674907081s) [noise=true , address=145.79.227.38:4979 , piece_id=725WBGV2ADSWWYKJL6DSK3WEXYCVJNFXWE4HZBX4JQMAX7FOZNQA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.803838363 (513h22m0.674921398s) [noise=true , address=145.79.227.15:4874 , piece_id=CXGQH5SYKXGKYUZNTSGC7IZLH4VWM2P3OMT57SNJ5T6TPFXZO76A, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.803883309 (513h22m0.674966354s) [noise=true , address=45.91.116.4:4937 , piece_id=UGEYXQXB5ZK6CSWWRGR7LTQL2CLOFTAB2IBPMQ4KKBSLZD4BNIYQ, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.804222103 (513h22m0.675305148s) [noise=true , address=145.79.227.39:4014 , piece_id=OQUFAAKDW4E7K2UAMHHEQ5ZFULJK66DT5AFGGPPPPD76MTWZI5FA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.804230931 (513h22m0.675313966s) [noise=true , address=77.90.31.10:4517 , piece_id=JFN2CPAUXOKQHCWS4KFHEOT46CLBYYSQFHTGNEFPOBO7LX54AI3A, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.804494342 (513h22m0.675577387s) [noise=true , address=145.79.227.37:4566 , piece_id=H6YR4MGSBZ4LC2RF6RKFAQJQDGHEMLIYSXVDO4S6ELMIXCUIKC2Q, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.804843817 (513h22m0.675926852s) [noise=true , address=77.90.31.21:4880 , piece_id=QW4UBY4XGQGBTUIG26DV5IDPVLVY5P7GFRWFRBORPBXKRBE2EPEA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.805175630 (513h22m0.676258675s) [noise=true , address=145.79.227.47:4265 , piece_id=AMEWXGKLCVMA3FNY443N3WUTKV3KOBZFSXWM44TVSCY3OVGHE4KQ, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.805520727 (513h22m0.676603762s) [noise=true , address=77.90.31.17:4770 , piece_id=XBLDSUF7Y5ZHLYQ3OYSC47QG5PQTJN545XUQDXSWAPFOREF3VC6Q, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.805915298 (513h22m0.676998343s) [noise=true , address=145.79.227.19:4077 , piece_id=QWIGWVSJIOHXIZZNFYQNCDCPEZCULFNBDBXTO2M4PHR2NSBGB6IA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.806382036 (513h22m0.677465071s) [noise=true , address=45.91.116.10:4082 , piece_id=IQO7KBQOJ2MSKPBEKQBGM7SF7R53MNOD6PP3J3OWMMM5P3RTR5JA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.806798419 (513h22m0.677881454s) [noise=true , address=77.90.31.11:4555 , piece_id=CUSNETFEN2NW3CNANNTVP6SOFMBDWU5IJTJL72INQIG4A4GXIT4Q, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.807040620 (513h22m0.678123655s) [noise=true , address=77.90.31.16:4735 , piece_id=MW7RYCJQKGXMGVILHOUWUDR6IMJC5NKYIRM6LS5STKZF2PSFADAQ, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.807343547 (513h22m0.678426582s) [noise=true , address=145.79.227.52:4445 , piece_id=77IW2FW5P7I543UY6BM7IDJEHRTA3NFHXCVJOA4NZP45UDERBOBA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.807543298 (513h22m0.678626333s) [noise=true , address=145.79.227.6:4257 , piece_id=O3IB62QVDQ7PISWKUTUI6HIF63BOLOISH3WC5KM6V4Y7PPZR7DHQ, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.807936917 (513h22m0.679019952s) [noise=true , address=145.79.227.51:4411 , piece_id=3INJQ25IWP4N7IUH3HPTVEZGCCEWZ5PKOK3UDNX54CMCORFSDVZQ, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:02.512919083 (513h22m2.384002128s) [upload=450474]: Next returned nil segment
Dec 27 00:51:02.512937648 (513h22m2.384020693s) [upload=450474]: Waiting for error group managing segments...
Dec 27 00:51:02.807570246 (513h22m2.678653282s) [noise=true , address=145.79.227.51:4411 , piece_id=3INJQ25IWP4N7IUH3HPTVEZGCCEWZ5PKOK3UDNX54CMCORFSDVZQ, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.808024241 (513h22m2.679107286s) [noise=true , address=145.79.227.37:4566 , piece_id=H6YR4MGSBZ4LC2RF6RKFAQJQDGHEMLIYSXVDO4S6ELMIXCUIKC2Q, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.808605929 (513h22m2.679688974s) [noise=true , address=145.79.227.15:4874 , piece_id=CXGQH5SYKXGKYUZNTSGC7IZLH4VWM2P3OMT57SNJ5T6TPFXZO76A, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.815238488 (513h22m2.686321533s) [noise=true , address=145.79.227.39:4014 , piece_id=OQUFAAKDW4E7K2UAMHHEQ5ZFULJK66DT5AFGGPPPPD76MTWZI5FA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.819686307 (513h22m2.690769353s) [noise=true , address=145.79.227.19:4077 , piece_id=QWIGWVSJIOHXIZZNFYQNCDCPEZCULFNBDBXTO2M4PHR2NSBGB6IA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.825441306 (513h22m2.696524351s) [noise=true , address=145.79.227.38:4979 , piece_id=725WBGV2ADSWWYKJL6DSK3WEXYCVJNFXWE4HZBX4JQMAX7FOZNQA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.828158389 (513h22m2.699241424s) [noise=true , address=145.79.227.47:4265 , piece_id=AMEWXGKLCVMA3FNY443N3WUTKV3KOBZFSXWM44TVSCY3OVGHE4KQ, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.832136715 (513h22m2.70321975s) [noise=true , address=145.79.227.52:4445 , piece_id=77IW2FW5P7I543UY6BM7IDJEHRTA3NFHXCVJOA4NZP45UDERBOBA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.976864034 (513h22m2.847947079s) [noise=true , address=77.90.31.17:4770 , piece_id=XBLDSUF7Y5ZHLYQ3OYSC47QG5PQTJN545XUQDXSWAPFOREF3VC6Q, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.986573411 (513h22m2.857656476s) [noise=true , address=77.90.31.21:4880 , piece_id=QW4UBY4XGQGBTUIG26DV5IDPVLVY5P7GFRWFRBORPBXKRBE2EPEA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.987637267 (513h22m2.858720333s) [noise=true , address=77.90.31.10:4517 , piece_id=JFN2CPAUXOKQHCWS4KFHEOT46CLBYYSQFHTGNEFPOBO7LX54AI3A, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:03.009321697 (513h22m2.880404742s) [noise=true , address=77.90.31.16:4735 , piece_id=MW7RYCJQKGXMGVILHOUWUDR6IMJC5NKYIRM6LS5STKZF2PSFADAQ, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:03.021098281 (513h22m2.892181316s) [noise=true , address=45.91.116.4:4937 , piece_id=UGEYXQXB5ZK6CSWWRGR7LTQL2CLOFTAB2IBPMQ4KKBSLZD4BNIYQ, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:03.021121575 (513h22m2.892204621s) [seg_pos={0 0}, upload=450474]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:51:03.021333629 (513h22m2.892416684s) [noise=true , address=45.91.116.10:4082 , piece_id=IQO7KBQOJ2MSKPBEKQBGM7SF7R53MNOD6PP3J3OWMMM5P3RTR5JA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 27 00:51:03.021343358 (513h22m2.892426393s) [noise=true , address=45.91.116.10:4082 , piece_id=IQO7KBQOJ2MSKPBEKQBGM7SF7R53MNOD6PP3J3OWMMM5P3RTR5JA, seg_pos={0 0}, upload=450474]: Long tail cancellation due to optimal threshold
Dec 27 00:51:03.021508833 (513h22m2.892591878s) [noise=true , address=77.90.31.11:4555 , piece_id=CUSNETFEN2NW3CNANNTVP6SOFMBDWU5IJTJL72INQIG4A4GXIT4Q, seg_pos={0 0}, upload=450474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Dec 27 00:51:03.021517880 (513h22m2.892600915s) [noise=true , address=77.90.31.11:4555 , piece_id=CUSNETFEN2NW3CNANNTVP6SOFMBDWU5IJTJL72INQIG4A4GXIT4Q, seg_pos={0 0}, upload=450474]: Long tail cancellation due to optimal threshold
Dec 27 00:51:03.021712370 (513h22m2.892795416s) [noise=true , address=145.79.227.6:4257 , piece_id=O3IB62QVDQ7PISWKUTUI6HIF63BOLOISH3WC5KM6V4Y7PPZR7DHQ, seg_pos={0 0}, upload=450474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:51:03.021721437 (513h22m2.892804473s) [noise=true , address=145.79.227.6:4257 , piece_id=O3IB62QVDQ7PISWKUTUI6HIF63BOLOISH3WC5KM6V4Y7PPZR7DHQ, seg_pos={0 0}, upload=450474]: Long tail cancellation due to optimal threshold
Dec 27 00:51:03.021739973 (513h22m2.892823018s) [seg_pos={0 0}, upload=450474]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:51:03.021886542 (513h22m2.892969577s) [upload=450474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:51:03.021893074 (513h22m2.892976109s) [upload=450474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:51:03.309303341 (513h22m3.180386376s) [upload=450474]: Done uploading segments...
Dec 27 00:51:03.309330002 (513h22m3.180413037s) [upload=450474]: Upload finished. err: <nil>
Duration: 6.561950346s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
11.336868122s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:51:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 11.336890795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X.X] duration:
30.002792169s trace:
https://jaeger.dev.storj.io/trace/2f2774bd1397dd6b
FAIL (2025-12-27 00:51:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: JFN2CPAUXOKQHCWS4KFHEOT46CLBYYSQFHTGNEFPOBO7LX54AI3A) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: QW4UBY4XGQGBTUIG26DV5IDPVLVY5P7GFRWFRBORPBXKRBE2EPEA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: XBLDSUF7Y5ZHLYQ3OYSC47QG5PQTJN545XUQDXSWAPFOREF3VC6Q) context deadline exceeded
======LOGS======
Dec 27 00:51:00.561417422 (513h22m0.432500467s) [upload=450474]: Starting upload
Dec 27 00:51:00.561475021 (513h22m0.432558056s) [upload=450474]: Done starting upload
Dec 27 00:51:00.561611852 (513h22m0.432694897s) [upload=450474]: Uploading segments...
Dec 27 00:51:00.561709558 (513h22m0.432792603s) [upload=450474]: Got next segment. Inline: false
Dec 27 00:51:00.561766978 (513h22m0.432850023s) [upload=450474]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:51:00.561773741 (513h22m0.432856786s) [upload=450474]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:51:00.803749764 (513h22m0.6748328s) [seg_pos={0 0}, upload=450474]: Begin upload segment...
Dec 27 00:51:00.803798527 (513h22m0.674881562s) [seg_pos={0 0}, upload=450474]: Done begin upload segment.
Dec 27 00:51:00.803824046 (513h22m0.674907081s) [noise=true , address=145.79.227.38:4979 , piece_id=725WBGV2ADSWWYKJL6DSK3WEXYCVJNFXWE4HZBX4JQMAX7FOZNQA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.803838363 (513h22m0.674921398s) [noise=true , address=145.79.227.15:4874 , piece_id=CXGQH5SYKXGKYUZNTSGC7IZLH4VWM2P3OMT57SNJ5T6TPFXZO76A, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.803883309 (513h22m0.674966354s) [noise=true , address=45.91.116.4:4937 , piece_id=UGEYXQXB5ZK6CSWWRGR7LTQL2CLOFTAB2IBPMQ4KKBSLZD4BNIYQ, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.804222103 (513h22m0.675305148s) [noise=true , address=145.79.227.39:4014 , piece_id=OQUFAAKDW4E7K2UAMHHEQ5ZFULJK66DT5AFGGPPPPD76MTWZI5FA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.804230931 (513h22m0.675313966s) [noise=true , address=77.90.31.10:4517 , piece_id=JFN2CPAUXOKQHCWS4KFHEOT46CLBYYSQFHTGNEFPOBO7LX54AI3A, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.804494342 (513h22m0.675577387s) [noise=true , address=145.79.227.37:4566 , piece_id=H6YR4MGSBZ4LC2RF6RKFAQJQDGHEMLIYSXVDO4S6ELMIXCUIKC2Q, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.804843817 (513h22m0.675926852s) [noise=true , address=77.90.31.21:4880 , piece_id=QW4UBY4XGQGBTUIG26DV5IDPVLVY5P7GFRWFRBORPBXKRBE2EPEA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.805175630 (513h22m0.676258675s) [noise=true , address=145.79.227.47:4265 , piece_id=AMEWXGKLCVMA3FNY443N3WUTKV3KOBZFSXWM44TVSCY3OVGHE4KQ, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.805520727 (513h22m0.676603762s) [noise=true , address=77.90.31.17:4770 , piece_id=XBLDSUF7Y5ZHLYQ3OYSC47QG5PQTJN545XUQDXSWAPFOREF3VC6Q, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.805915298 (513h22m0.676998343s) [noise=true , address=145.79.227.19:4077 , piece_id=QWIGWVSJIOHXIZZNFYQNCDCPEZCULFNBDBXTO2M4PHR2NSBGB6IA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.806382036 (513h22m0.677465071s) [noise=true , address=45.91.116.10:4082 , piece_id=IQO7KBQOJ2MSKPBEKQBGM7SF7R53MNOD6PP3J3OWMMM5P3RTR5JA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.806798419 (513h22m0.677881454s) [noise=true , address=77.90.31.11:4555 , piece_id=CUSNETFEN2NW3CNANNTVP6SOFMBDWU5IJTJL72INQIG4A4GXIT4Q, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.807040620 (513h22m0.678123655s) [noise=true , address=77.90.31.16:4735 , piece_id=MW7RYCJQKGXMGVILHOUWUDR6IMJC5NKYIRM6LS5STKZF2PSFADAQ, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.807343547 (513h22m0.678426582s) [noise=true , address=145.79.227.52:4445 , piece_id=77IW2FW5P7I543UY6BM7IDJEHRTA3NFHXCVJOA4NZP45UDERBOBA, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.807543298 (513h22m0.678626333s) [noise=true , address=145.79.227.6:4257 , piece_id=O3IB62QVDQ7PISWKUTUI6HIF63BOLOISH3WC5KM6V4Y7PPZR7DHQ, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:00.807936917 (513h22m0.679019952s) [noise=true , address=145.79.227.51:4411 , piece_id=3INJQ25IWP4N7IUH3HPTVEZGCCEWZ5PKOK3UDNX54CMCORFSDVZQ, seg_pos={0 0}, upload=450474]: Uploading piece...
Dec 27 00:51:02.512919083 (513h22m2.384002128s) [upload=450474]: Next returned nil segment
Dec 27 00:51:02.512937648 (513h22m2.384020693s) [upload=450474]: Waiting for error group managing segments...
Dec 27 00:51:02.807570246 (513h22m2.678653282s) [noise=true , address=145.79.227.51:4411 , piece_id=3INJQ25IWP4N7IUH3HPTVEZGCCEWZ5PKOK3UDNX54CMCORFSDVZQ, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.808024241 (513h22m2.679107286s) [noise=true , address=145.79.227.37:4566 , piece_id=H6YR4MGSBZ4LC2RF6RKFAQJQDGHEMLIYSXVDO4S6ELMIXCUIKC2Q, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.808605929 (513h22m2.679688974s) [noise=true , address=145.79.227.15:4874 , piece_id=CXGQH5SYKXGKYUZNTSGC7IZLH4VWM2P3OMT57SNJ5T6TPFXZO76A, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.815238488 (513h22m2.686321533s) [noise=true , address=145.79.227.39:4014 , piece_id=OQUFAAKDW4E7K2UAMHHEQ5ZFULJK66DT5AFGGPPPPD76MTWZI5FA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.819686307 (513h22m2.690769353s) [noise=true , address=145.79.227.19:4077 , piece_id=QWIGWVSJIOHXIZZNFYQNCDCPEZCULFNBDBXTO2M4PHR2NSBGB6IA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.825441306 (513h22m2.696524351s) [noise=true , address=145.79.227.38:4979 , piece_id=725WBGV2ADSWWYKJL6DSK3WEXYCVJNFXWE4HZBX4JQMAX7FOZNQA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.828158389 (513h22m2.699241424s) [noise=true , address=145.79.227.47:4265 , piece_id=AMEWXGKLCVMA3FNY443N3WUTKV3KOBZFSXWM44TVSCY3OVGHE4KQ, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.832136715 (513h22m2.70321975s) [noise=true , address=145.79.227.52:4445 , piece_id=77IW2FW5P7I543UY6BM7IDJEHRTA3NFHXCVJOA4NZP45UDERBOBA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.976864034 (513h22m2.847947079s) [noise=true , address=77.90.31.17:4770 , piece_id=XBLDSUF7Y5ZHLYQ3OYSC47QG5PQTJN545XUQDXSWAPFOREF3VC6Q, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.986573411 (513h22m2.857656476s) [noise=true , address=77.90.31.21:4880 , piece_id=QW4UBY4XGQGBTUIG26DV5IDPVLVY5P7GFRWFRBORPBXKRBE2EPEA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:02.987637267 (513h22m2.858720333s) [noise=true , address=77.90.31.10:4517 , piece_id=JFN2CPAUXOKQHCWS4KFHEOT46CLBYYSQFHTGNEFPOBO7LX54AI3A, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:03.009321697 (513h22m2.880404742s) [noise=true , address=77.90.31.16:4735 , piece_id=MW7RYCJQKGXMGVILHOUWUDR6IMJC5NKYIRM6LS5STKZF2PSFADAQ, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:03.021098281 (513h22m2.892181316s) [noise=true , address=45.91.116.4:4937 , piece_id=UGEYXQXB5ZK6CSWWRGR7LTQL2CLOFTAB2IBPMQ4KKBSLZD4BNIYQ, seg_pos={0 0}, upload=450474]: Done uploading piece. err: <nil>
Dec 27 00:51:03.021121575 (513h22m2.892204621s) [seg_pos={0 0}, upload=450474]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:51:03.021333629 (513h22m2.892416684s) [noise=true , address=45.91.116.10:4082 , piece_id=IQO7KBQOJ2MSKPBEKQBGM7SF7R53MNOD6PP3J3OWMMM5P3RTR5JA, seg_pos={0 0}, upload=450474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF
Dec 27 00:51:03.021343358 (513h22m2.892426393s) [noise=true , address=45.91.116.10:4082 , piece_id=IQO7KBQOJ2MSKPBEKQBGM7SF7R53MNOD6PP3J3OWMMM5P3RTR5JA, seg_pos={0 0}, upload=450474]: Long tail cancellation due to optimal threshold
Dec 27 00:51:03.021508833 (513h22m2.892591878s) [noise=true , address=77.90.31.11:4555 , piece_id=CUSNETFEN2NW3CNANNTVP6SOFMBDWU5IJTJL72INQIG4A4GXIT4Q, seg_pos={0 0}, upload=450474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Dec 27 00:51:03.021517880 (513h22m2.892600915s) [noise=true , address=77.90.31.11:4555 , piece_id=CUSNETFEN2NW3CNANNTVP6SOFMBDWU5IJTJL72INQIG4A4GXIT4Q, seg_pos={0 0}, upload=450474]: Long tail cancellation due to optimal threshold
Dec 27 00:51:03.021712370 (513h22m2.892795416s) [noise=true , address=145.79.227.6:4257 , piece_id=O3IB62QVDQ7PISWKUTUI6HIF63BOLOISH3WC5KM6V4Y7PPZR7DHQ, seg_pos={0 0}, upload=450474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:51:03.021721437 (513h22m2.892804473s) [noise=true , address=145.79.227.6:4257 , piece_id=O3IB62QVDQ7PISWKUTUI6HIF63BOLOISH3WC5KM6V4Y7PPZR7DHQ, seg_pos={0 0}, upload=450474]: Long tail cancellation due to optimal threshold
Dec 27 00:51:03.021739973 (513h22m2.892823018s) [seg_pos={0 0}, upload=450474]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:51:03.021886542 (513h22m2.892969577s) [upload=450474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:51:03.021893074 (513h22m2.892976109s) [upload=450474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:51:03.309303341 (513h22m3.180386376s) [upload=450474]: Done uploading segments...
Dec 27 00:51:03.309330002 (513h22m3.180413037s) [upload=450474]: Upload finished. err: <nil>
Duration: 30.002820983s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
22.374217928s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:51:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 22.374251072s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
22.195358459s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:50:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 22.195390418s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
26 out of 60 is failed (threshold=3) [.X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X.] duration:
6.104236193s trace:
https://jaeger.dev.storj.io/trace/7aac3b5e78d3d16f
FAIL (2025-12-27 00:50:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
lse
Dec 27 00:49:27.191150506 (513h20m27.062233551s) [upload=450449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:49:27.191156648 (513h20m27.062239683s) [upload=450449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:49:27.436801067 (513h20m27.307884102s) [seg_pos={0 0}, upload=450449]: Begin upload segment...
Dec 27 00:49:27.436853677 (513h20m27.307936712s) [seg_pos={0 0}, upload=450449]: Done begin upload segment.
Dec 27 00:49:27.436884336 (513h20m27.307967371s) [noise=true , address=77.90.31.11:4554 , piece_id=LTEJRLYELIPX5SV5O7LGOB5KIEV2UIH6HLOSBVJU7GXNGZIWZM5Q, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.436919403 (513h20m27.308002438s) [noise=true , address=45.91.116.11:4118 , piece_id=NX36QVRSNHN52SH22M2EDXSI2VCF5HIWLGOONDUYBKXLMJHEKGAA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.437334263 (513h20m27.308417308s) [noise=true , address=145.79.227.11:4730 , piece_id=GPKDWWDYHQC52YLBSWFONCXKU72TSGSI4MPZUBAG6RQYOQKR44GA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.437353789 (513h20m27.308436824s) [noise=true , address=145.79.227.4:4185 , piece_id=QSZKVYIXS2DSFBEOCWYMKCFKZURGJF4VXEUK7JQRQMVY7YDOKM2Q, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.437631027 (513h20m27.308714062s) [noise=true , address=45.91.116.5:4300 , piece_id=2MRMYF3P7R66NSKX7KUMHO4Q3C3HBTXYQ55LMRO47YN5S2KU7BHA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.437708495 (513h20m27.30879154s) [noise=true , address=145.79.227.15:4875 , piece_id=SAWWQ53WBUAWSFYIFKZKNSWHTQKOXWQ226KXJP2Q7L6NKCWOMGAA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.437911281 (513h20m27.308994316s) [noise=true , address=45.91.116.8:4048 , piece_id=EYJZ7AD2XSLOTTP7LJUBMDDVZ3U4TRZEJ43NHCI3QG6MZB3OJ7XQ, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.438051919 (513h20m27.309134954s) [noise=true , address=145.79.227.39:4014 , piece_id=KT3ENHETOTCMT4D53Y4BUUADSAJDBBQKML2G5R2A2EMEBOROETGQ, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.438286455 (513h20m27.309369501s) [noise=true , address=77.90.31.15:4700 , piece_id=XVBCABM5L3Z6HVFSODAFI5KQVSXSYJIVSWL3V4MKLPYYCQADPQNA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.438437833 (513h20m27.309520868s) [noise=true , address=77.90.31.18:4808 , piece_id=AJEPDBWAVY35LAU2POV5FMPBPOXCXLQFP3HK3JGUUMD3I6UWFHJA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.438551240 (513h20m27.309634275s) [noise=true , address=145.79.227.38:4977 , piece_id=ZVA3ZMTUAVVRH4CX52MMUHJ62ID3NM52V5FZGUI5FRMCPTLHMHBA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.438781488 (513h20m27.309864523s) [noise=true , address=145.79.227.50:4374 , piece_id=4IUTBR2VDI5YEZVEFE3RS4UZ6ZZYC34R4MIDXORFAALNCGM2WIMQ, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.439186580 (513h20m27.310269615s) [noise=true , address=45.91.116.14:4192 , piece_id=YF4Z6MLHG4TUVYSBCN5UIFEV2BB34MLUHCK3ECQTMB6OBNFJUSLA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.439508532 (513h20m27.310591567s) [noise=true , address=145.79.227.52:4446 , piece_id=UYYA7Y327OD3AVVTIKDBFJH3XCYFKBBQADB3ALW44E2LG3M6B7HA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.440006971 (513h20m27.311090016s) [noise=true , address=145.79.227.37:4566 , piece_id=RU6NU3JA2YOBKNXVD46H4TRPJK5LCYGILHPYAQDTFE2BNP7Q7JXQ, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.440827264 (513h20m27.311910299s) [noise=true , address=145.79.227.41:4087 , piece_id=WRAQWJLT2QGMKOVOM6GDAWIILQRE52MIX4LNHOEPASWVKDLDZ7ZA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:29.143112693 (513h20m29.014195738s) [upload=450449]: Next returned nil segment
Dec 27 00:49:29.143163620 (513h20m29.014246676s) [upload=450449]: Waiting for error group managing segments...
Dec 27 00:49:29.429103858 (513h20m29.300186903s) [noise=true , address=145.79.227.50:4374 , piece_id=4IUTBR2VDI5YEZVEFE3RS4UZ6ZZYC34R4MIDXORFAALNCGM2WIMQ, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.435345984 (513h20m29.306429019s) [noise=true , address=145.79.227.11:4730 , piece_id=GPKDWWDYHQC52YLBSWFONCXKU72TSGSI4MPZUBAG6RQYOQKR44GA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.443852080 (513h20m29.314935115s) [noise=true , address=145.79.227.37:4566 , piece_id=RU6NU3JA2YOBKNXVD46H4TRPJK5LCYGILHPYAQDTFE2BNP7Q7JXQ, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.447956505 (513h20m29.31903954s) [noise=true , address=145.79.227.52:4446 , piece_id=UYYA7Y327OD3AVVTIKDBFJH3XCYFKBBQADB3ALW44E2LG3M6B7HA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.448714979 (513h20m29.319798015s) [noise=true , address=145.79.227.41:4087 , piece_id=WRAQWJLT2QGMKOVOM6GDAWIILQRE52MIX4LNHOEPASWVKDLDZ7ZA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.451308838 (513h20m29.322391883s) [noise=true , address=145.79.227.38:4977 , piece_id=ZVA3ZMTUAVVRH4CX52MMUHJ62ID3NM52V5FZGUI5FRMCPTLHMHBA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.458592056 (513h20m29.329675101s) [noise=true , address=145.79.227.15:4875 , piece_id=SAWWQ53WBUAWSFYIFKZKNSWHTQKOXWQ226KXJP2Q7L6NKCWOMGAA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.479079778 (513h20m29.350162823s) [noise=true , address=145.79.227.39:4014 , piece_id=KT3ENHETOTCMT4D53Y4BUUADSAJDBBQKML2G5R2A2EMEBOROETGQ, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.608465085 (513h20m29.47954812s) [noise=true , address=77.90.31.18:4808 , piece_id=AJEPDBWAVY35LAU2POV5FMPBPOXCXLQFP3HK3JGUUMD3I6UWFHJA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.634034553 (513h20m29.505117588s) [noise=true , address=77.90.31.15:4700 , piece_id=XVBCABM5L3Z6HVFSODAFI5KQVSXSYJIVSWL3V4MKLPYYCQADPQNA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.765223985 (513h20m29.63630703s) [noise=true , address=45.91.116.5:4300 , piece_id=2MRMYF3P7R66NSKX7KUMHO4Q3C3HBTXYQ55LMRO47YN5S2KU7BHA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.817847036 (513h20m29.688930071s) [noise=true , address=45.91.116.14:4192 , piece_id=YF4Z6MLHG4TUVYSBCN5UIFEV2BB34MLUHCK3ECQTMB6OBNFJUSLA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.897336514 (513h20m29.768419559s) [noise=true , address=45.91.116.8:4048 , piece_id=EYJZ7AD2XSLOTTP7LJUBMDDVZ3U4TRZEJ43NHCI3QG6MZB3OJ7XQ, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.897360739 (513h20m29.768443774s) [seg_pos={0 0}, upload=450449]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:49:29.897651684 (513h20m29.768734739s) [noise=true , address=77.90.31.11:4554 , piece_id=LTEJRLYELIPX5SV5O7LGOB5KIEV2UIH6HLOSBVJU7GXNGZIWZM5Q, seg_pos={0 0}, upload=450449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:35988->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:35988->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35988->77.90.31.11:4554: use of closed network connection
Dec 27 00:49:29.897662174 (513h20m29.768745209s) [noise=true , address=77.90.31.11:4554 , piece_id=LTEJRLYELIPX5SV5O7LGOB5KIEV2UIH6HLOSBVJU7GXNGZIWZM5Q, seg_pos={0 0}, upload=450449]: Long tail cancellation due to optimal threshold
Dec 27 00:49:29.897750732 (513h20m29.768833767s) [noise=true , address=145.79.227.4:4185 , piece_id=QSZKVYIXS2DSFBEOCWYMKCFKZURGJF4VXEUK7JQRQMVY7YDOKM2Q, seg_pos={0 0}, upload=450449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:54548->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:54548->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54548->145.79.227.4:4185: use of closed network connection
Dec 27 00:49:29.897767113 (513h20m29.768850149s) [noise=true , address=145.79.227.4:4185 , piece_id=QSZKVYIXS2DSFBEOCWYMKCFKZURGJF4VXEUK7JQRQMVY7YDOKM2Q, seg_pos={0 0}, upload=450449]: Long tail cancellation due to optimal threshold
Dec 27 00:49:29.898039072 (513h20m29.769122107s) [noise=true , address=45.91.116.11:4118 , piece_id=NX36QVRSNHN52SH22M2EDXSI2VCF5HIWLGOONDUYBKXLMJHEKGAA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: read tcp 188.166.225.170:36608->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 00:49:29.898048730 (513h20m29.769131775s) [noise=true , address=45.91.116.11:4118 , piece_id=NX36QVRSNHN52SH22M2EDXSI2VCF5HIWLGOONDUYBKXLMJHEKGAA, seg_pos={0 0}, upload=450449]: Long tail cancellation due to optimal threshold
Dec 27 00:49:29.898063618 (513h20m29.769146663s) [seg_pos={0 0}, upload=450449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:49:29.898143060 (513h20m29.769226095s) [upload=450449]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:49:29.898148400 (513h20m29.769231435s) [upload=450449]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:49:30.197951954 (513h20m30.069034999s) [upload=450449]: Done uploading segments...
Dec 27 00:49:30.197978625 (513h20m30.06906166s) [upload=450449]: Upload finished. err: <nil>
Duration: 6.104259687s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
26 out of 60 is failed (threshold=3) [..X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...X] duration:
30.001963236s trace:
https://jaeger.dev.storj.io/trace/7aac3b5e78d3d16f
FAIL (2025-12-27 00:49:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
lse
Dec 27 00:49:27.191150506 (513h20m27.062233551s) [upload=450449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:49:27.191156648 (513h20m27.062239683s) [upload=450449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:49:27.436801067 (513h20m27.307884102s) [seg_pos={0 0}, upload=450449]: Begin upload segment...
Dec 27 00:49:27.436853677 (513h20m27.307936712s) [seg_pos={0 0}, upload=450449]: Done begin upload segment.
Dec 27 00:49:27.436884336 (513h20m27.307967371s) [noise=true , address=77.90.31.11:4554 , piece_id=LTEJRLYELIPX5SV5O7LGOB5KIEV2UIH6HLOSBVJU7GXNGZIWZM5Q, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.436919403 (513h20m27.308002438s) [noise=true , address=45.91.116.11:4118 , piece_id=NX36QVRSNHN52SH22M2EDXSI2VCF5HIWLGOONDUYBKXLMJHEKGAA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.437334263 (513h20m27.308417308s) [noise=true , address=145.79.227.11:4730 , piece_id=GPKDWWDYHQC52YLBSWFONCXKU72TSGSI4MPZUBAG6RQYOQKR44GA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.437353789 (513h20m27.308436824s) [noise=true , address=145.79.227.4:4185 , piece_id=QSZKVYIXS2DSFBEOCWYMKCFKZURGJF4VXEUK7JQRQMVY7YDOKM2Q, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.437631027 (513h20m27.308714062s) [noise=true , address=45.91.116.5:4300 , piece_id=2MRMYF3P7R66NSKX7KUMHO4Q3C3HBTXYQ55LMRO47YN5S2KU7BHA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.437708495 (513h20m27.30879154s) [noise=true , address=145.79.227.15:4875 , piece_id=SAWWQ53WBUAWSFYIFKZKNSWHTQKOXWQ226KXJP2Q7L6NKCWOMGAA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.437911281 (513h20m27.308994316s) [noise=true , address=45.91.116.8:4048 , piece_id=EYJZ7AD2XSLOTTP7LJUBMDDVZ3U4TRZEJ43NHCI3QG6MZB3OJ7XQ, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.438051919 (513h20m27.309134954s) [noise=true , address=145.79.227.39:4014 , piece_id=KT3ENHETOTCMT4D53Y4BUUADSAJDBBQKML2G5R2A2EMEBOROETGQ, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.438286455 (513h20m27.309369501s) [noise=true , address=77.90.31.15:4700 , piece_id=XVBCABM5L3Z6HVFSODAFI5KQVSXSYJIVSWL3V4MKLPYYCQADPQNA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.438437833 (513h20m27.309520868s) [noise=true , address=77.90.31.18:4808 , piece_id=AJEPDBWAVY35LAU2POV5FMPBPOXCXLQFP3HK3JGUUMD3I6UWFHJA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.438551240 (513h20m27.309634275s) [noise=true , address=145.79.227.38:4977 , piece_id=ZVA3ZMTUAVVRH4CX52MMUHJ62ID3NM52V5FZGUI5FRMCPTLHMHBA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.438781488 (513h20m27.309864523s) [noise=true , address=145.79.227.50:4374 , piece_id=4IUTBR2VDI5YEZVEFE3RS4UZ6ZZYC34R4MIDXORFAALNCGM2WIMQ, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.439186580 (513h20m27.310269615s) [noise=true , address=45.91.116.14:4192 , piece_id=YF4Z6MLHG4TUVYSBCN5UIFEV2BB34MLUHCK3ECQTMB6OBNFJUSLA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.439508532 (513h20m27.310591567s) [noise=true , address=145.79.227.52:4446 , piece_id=UYYA7Y327OD3AVVTIKDBFJH3XCYFKBBQADB3ALW44E2LG3M6B7HA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.440006971 (513h20m27.311090016s) [noise=true , address=145.79.227.37:4566 , piece_id=RU6NU3JA2YOBKNXVD46H4TRPJK5LCYGILHPYAQDTFE2BNP7Q7JXQ, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:27.440827264 (513h20m27.311910299s) [noise=true , address=145.79.227.41:4087 , piece_id=WRAQWJLT2QGMKOVOM6GDAWIILQRE52MIX4LNHOEPASWVKDLDZ7ZA, seg_pos={0 0}, upload=450449]: Uploading piece...
Dec 27 00:49:29.143112693 (513h20m29.014195738s) [upload=450449]: Next returned nil segment
Dec 27 00:49:29.143163620 (513h20m29.014246676s) [upload=450449]: Waiting for error group managing segments...
Dec 27 00:49:29.429103858 (513h20m29.300186903s) [noise=true , address=145.79.227.50:4374 , piece_id=4IUTBR2VDI5YEZVEFE3RS4UZ6ZZYC34R4MIDXORFAALNCGM2WIMQ, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.435345984 (513h20m29.306429019s) [noise=true , address=145.79.227.11:4730 , piece_id=GPKDWWDYHQC52YLBSWFONCXKU72TSGSI4MPZUBAG6RQYOQKR44GA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.443852080 (513h20m29.314935115s) [noise=true , address=145.79.227.37:4566 , piece_id=RU6NU3JA2YOBKNXVD46H4TRPJK5LCYGILHPYAQDTFE2BNP7Q7JXQ, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.447956505 (513h20m29.31903954s) [noise=true , address=145.79.227.52:4446 , piece_id=UYYA7Y327OD3AVVTIKDBFJH3XCYFKBBQADB3ALW44E2LG3M6B7HA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.448714979 (513h20m29.319798015s) [noise=true , address=145.79.227.41:4087 , piece_id=WRAQWJLT2QGMKOVOM6GDAWIILQRE52MIX4LNHOEPASWVKDLDZ7ZA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.451308838 (513h20m29.322391883s) [noise=true , address=145.79.227.38:4977 , piece_id=ZVA3ZMTUAVVRH4CX52MMUHJ62ID3NM52V5FZGUI5FRMCPTLHMHBA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.458592056 (513h20m29.329675101s) [noise=true , address=145.79.227.15:4875 , piece_id=SAWWQ53WBUAWSFYIFKZKNSWHTQKOXWQ226KXJP2Q7L6NKCWOMGAA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.479079778 (513h20m29.350162823s) [noise=true , address=145.79.227.39:4014 , piece_id=KT3ENHETOTCMT4D53Y4BUUADSAJDBBQKML2G5R2A2EMEBOROETGQ, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.608465085 (513h20m29.47954812s) [noise=true , address=77.90.31.18:4808 , piece_id=AJEPDBWAVY35LAU2POV5FMPBPOXCXLQFP3HK3JGUUMD3I6UWFHJA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.634034553 (513h20m29.505117588s) [noise=true , address=77.90.31.15:4700 , piece_id=XVBCABM5L3Z6HVFSODAFI5KQVSXSYJIVSWL3V4MKLPYYCQADPQNA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.765223985 (513h20m29.63630703s) [noise=true , address=45.91.116.5:4300 , piece_id=2MRMYF3P7R66NSKX7KUMHO4Q3C3HBTXYQ55LMRO47YN5S2KU7BHA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.817847036 (513h20m29.688930071s) [noise=true , address=45.91.116.14:4192 , piece_id=YF4Z6MLHG4TUVYSBCN5UIFEV2BB34MLUHCK3ECQTMB6OBNFJUSLA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.897336514 (513h20m29.768419559s) [noise=true , address=45.91.116.8:4048 , piece_id=EYJZ7AD2XSLOTTP7LJUBMDDVZ3U4TRZEJ43NHCI3QG6MZB3OJ7XQ, seg_pos={0 0}, upload=450449]: Done uploading piece. err: <nil>
Dec 27 00:49:29.897360739 (513h20m29.768443774s) [seg_pos={0 0}, upload=450449]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:49:29.897651684 (513h20m29.768734739s) [noise=true , address=77.90.31.11:4554 , piece_id=LTEJRLYELIPX5SV5O7LGOB5KIEV2UIH6HLOSBVJU7GXNGZIWZM5Q, seg_pos={0 0}, upload=450449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:35988->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:35988->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35988->77.90.31.11:4554: use of closed network connection
Dec 27 00:49:29.897662174 (513h20m29.768745209s) [noise=true , address=77.90.31.11:4554 , piece_id=LTEJRLYELIPX5SV5O7LGOB5KIEV2UIH6HLOSBVJU7GXNGZIWZM5Q, seg_pos={0 0}, upload=450449]: Long tail cancellation due to optimal threshold
Dec 27 00:49:29.897750732 (513h20m29.768833767s) [noise=true , address=145.79.227.4:4185 , piece_id=QSZKVYIXS2DSFBEOCWYMKCFKZURGJF4VXEUK7JQRQMVY7YDOKM2Q, seg_pos={0 0}, upload=450449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:54548->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:54548->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54548->145.79.227.4:4185: use of closed network connection
Dec 27 00:49:29.897767113 (513h20m29.768850149s) [noise=true , address=145.79.227.4:4185 , piece_id=QSZKVYIXS2DSFBEOCWYMKCFKZURGJF4VXEUK7JQRQMVY7YDOKM2Q, seg_pos={0 0}, upload=450449]: Long tail cancellation due to optimal threshold
Dec 27 00:49:29.898039072 (513h20m29.769122107s) [noise=true , address=45.91.116.11:4118 , piece_id=NX36QVRSNHN52SH22M2EDXSI2VCF5HIWLGOONDUYBKXLMJHEKGAA, seg_pos={0 0}, upload=450449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: read tcp 188.166.225.170:36608->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 00:49:29.898048730 (513h20m29.769131775s) [noise=true , address=45.91.116.11:4118 , piece_id=NX36QVRSNHN52SH22M2EDXSI2VCF5HIWLGOONDUYBKXLMJHEKGAA, seg_pos={0 0}, upload=450449]: Long tail cancellation due to optimal threshold
Dec 27 00:49:29.898063618 (513h20m29.769146663s) [seg_pos={0 0}, upload=450449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:49:29.898143060 (513h20m29.769226095s) [upload=450449]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:49:29.898148400 (513h20m29.769231435s) [upload=450449]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:49:30.197951954 (513h20m30.069034999s) [upload=450449]: Done uploading segments...
Dec 27 00:49:30.197978625 (513h20m30.06906166s) [upload=450449]: Upload finished. err: <nil>
Duration: 30.001996329s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
21.255705475s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:49:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 21.25572902s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [...X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X...] duration:
9.589237635s trace:
https://jaeger.dev.storj.io/trace/613b4980e2af9b90
FAIL (2025-12-27 00:49:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A) context deadline exceeded
======LOGS======
ec 27 00:46:53.091927718 (513h17m52.963010763s) [upload=450410]: Starting upload
Dec 27 00:46:53.092014754 (513h17m52.963097789s) [upload=450410]: Done starting upload
Dec 27 00:46:53.092086631 (513h17m52.963169676s) [upload=450410]: Uploading segments...
Dec 27 00:46:53.092246345 (513h17m52.96332939s) [upload=450410]: Got next segment. Inline: false
Dec 27 00:46:53.092289137 (513h17m52.963372182s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:46:53.092294687 (513h17m52.963377732s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:46:53.338382169 (513h17m53.209465224s) [seg_pos={0 0}, upload=450410]: Begin upload segment...
Dec 27 00:46:53.338449107 (513h17m53.209532142s) [seg_pos={0 0}, upload=450410]: Done begin upload segment.
Dec 27 00:46:53.338502960 (513h17m53.209586005s) [noise=true , address=45.91.116.6:4973 , piece_id=IN4XC2IM7CMYJOUYOOUQHGPVZFNPQRKJ7RGVAJEWXA43RFKBSZZQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.338571059 (513h17m53.209654104s) [noise=true , address=77.90.31.16:4735 , piece_id=G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339312621 (513h17m53.210395656s) [noise=true , address=145.79.227.48:4302 , piece_id=FIQELWLIL3WTQ3GCZPYXRWZX6CJC5JJ2ERPMB4ACWEXRTCTKFBKQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339380901 (513h17m53.210463946s) [noise=true , address=77.90.31.11:4555 , piece_id=K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339792144 (513h17m53.210875189s) [noise=true , address=145.79.227.13:4804 , piece_id=G43OXV7NHD6HLEDOX4PDUEZSVKEL26PEVQ67HRRTJAA3DFIB6G3A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340132612 (513h17m53.211215647s) [noise=true , address=145.79.227.42:4121 , piece_id=CPPL66BYG3N3CVKR423TWTQTAGPHBSOJHNUWV44O2VD3NAQ4BFCQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340504771 (513h17m53.211587806s) [noise=true , address=145.79.227.50:4375 , piece_id=BCOBU74QR2CAJDGTHJRLXHGVYTLNEE27XB2ASUOMCQE3CYB5GASQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340841783 (513h17m53.211924828s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.341158887 (513h17m53.212241922s) [noise=true , address=145.79.227.52:4445 , piece_id=FKJJIBP4GUIEDRCNVZ6D5NHVMLGEZEYLBHLLVGREFNOBUPIUFIZA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.341439952 (513h17m53.212522997s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.342547922 (513h17m53.213630967s) [noise=true , address=145.79.227.16:4911 , piece_id=EMBD4H7WYYQP4OF5S7XI6RVZSPB2E2ZUJFRJYL55MOJMIHSXV2VA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343041732 (513h17m53.214124777s) [noise=true , address=145.79.227.3:4149 , piece_id=YBMTYYTTCUSD7OYCT7UYBD5DJ45OPJFUUQL7LZCZUTFTGEIEXRPQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343522357 (513h17m53.214605402s) [noise=true , address=145.79.227.51:4409 , piece_id=B6ILY6YECF6AKPPTK25MXH3V42UPW5PT26BRSEOGR5ZDCKCQWVYQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343608741 (513h17m53.214691786s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343960280 (513h17m53.215043316s) [noise=true , address=145.79.227.41:4086 , piece_id=HQZRCBSE3KVF7WXODTR7TQLC6DLFO7Z7P442KYGEEFKEQEDD24UA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.344310978 (513h17m53.215394023s) [noise=true , address=145.79.227.15:4875 , piece_id=H6FVGFPL52MLHHLMF4GAEH5IWLZAP7UO25YZXYSIVK4NUDYL3WOA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:55.045381986 (513h17m54.916465031s) [upload=450410]: Next returned nil segment
Dec 27 00:46:55.045399729 (513h17m54.916482794s) [upload=450410]: Waiting for error group managing segments...
Dec 27 00:46:55.335282585 (513h17m55.20636562s) [noise=true , address=145.79.227.50:4375 , piece_id=BCOBU74QR2CAJDGTHJRLXHGVYTLNEE27XB2ASUOMCQE3CYB5GASQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.338011361 (513h17m55.209094396s) [noise=true , address=145.79.227.3:4149 , piece_id=YBMTYYTTCUSD7OYCT7UYBD5DJ45OPJFUUQL7LZCZUTFTGEIEXRPQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.339463746 (513h17m55.210546791s) [noise=true , address=145.79.227.48:4302 , piece_id=FIQELWLIL3WTQ3GCZPYXRWZX6CJC5JJ2ERPMB4ACWEXRTCTKFBKQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.340722183 (513h17m55.211805228s) [noise=true , address=145.79.227.15:4875 , piece_id=H6FVGFPL52MLHHLMF4GAEH5IWLZAP7UO25YZXYSIVK4NUDYL3WOA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.343886809 (513h17m55.214969844s) [noise=true , address=145.79.227.51:4409 , piece_id=B6ILY6YECF6AKPPTK25MXH3V42UPW5PT26BRSEOGR5ZDCKCQWVYQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.349629133 (513h17m55.220712168s) [noise=true , address=145.79.227.52:4445 , piece_id=FKJJIBP4GUIEDRCNVZ6D5NHVMLGEZEYLBHLLVGREFNOBUPIUFIZA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.353120751 (513h17m55.224203786s) [noise=true , address=145.79.227.13:4804 , piece_id=G43OXV7NHD6HLEDOX4PDUEZSVKEL26PEVQ67HRRTJAA3DFIB6G3A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.357296662 (513h17m55.228379707s) [noise=true , address=145.79.227.16:4911 , piece_id=EMBD4H7WYYQP4OF5S7XI6RVZSPB2E2ZUJFRJYL55MOJMIHSXV2VA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.368751163 (513h17m55.239834208s) [noise=true , address=145.79.227.42:4121 , piece_id=CPPL66BYG3N3CVKR423TWTQTAGPHBSOJHNUWV44O2VD3NAQ4BFCQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.369848654 (513h17m55.240931689s) [noise=true , address=145.79.227.41:4086 , piece_id=HQZRCBSE3KVF7WXODTR7TQLC6DLFO7Z7P442KYGEEFKEQEDD24UA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.503905865 (513h17m55.3749889s) [noise=true , address=77.90.31.11:4555 , piece_id=K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.544674503 (513h17m55.415757548s) [noise=true , address=77.90.31.16:4735 , piece_id=G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.583254845 (513h17m55.45433788s) [noise=true , address=45.91.116.6:4973 , piece_id=IN4XC2IM7CMYJOUYOOUQHGPVZFNPQRKJ7RGVAJEWXA43RFKBSZZQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.583272228 (513h17m55.454355263s) [seg_pos={0 0}, upload=450410]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:46:55.583448484 (513h17m55.454531519s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection
Dec 27 00:46:55.583454526 (513h17m55.454537561s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583729449 (513h17m55.454812504s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 27 00:46:55.583748826 (513h17m55.454831881s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583765217 (513h17m55.454848252s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:46:55.583774265 (513h17m55.4548573s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583988202 (513h17m55.455071247s) [seg_pos={0 0}, upload=450410]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:46:55.584230393 (513h17m55.455313428s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:46:55.584253126 (513h17m55.455336171s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:46:55.883178679 (513h17m55.754261714s) [upload=450410]: Done uploading segments...
Dec 27 00:46:55.883199319 (513h17m55.754282354s) [upload=450410]: Upload finished. err: <nil>
Duration: 9.589263804s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
26.498631281s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:49:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 26.498655859s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [....X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X..] duration:
21.648803877s trace:
https://jaeger.dev.storj.io/trace/613b4980e2af9b90
FAIL (2025-12-27 00:48:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A) context deadline exceeded
======LOGS======
ec 27 00:46:53.091927718 (513h17m52.963010763s) [upload=450410]: Starting upload
Dec 27 00:46:53.092014754 (513h17m52.963097789s) [upload=450410]: Done starting upload
Dec 27 00:46:53.092086631 (513h17m52.963169676s) [upload=450410]: Uploading segments...
Dec 27 00:46:53.092246345 (513h17m52.96332939s) [upload=450410]: Got next segment. Inline: false
Dec 27 00:46:53.092289137 (513h17m52.963372182s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:46:53.092294687 (513h17m52.963377732s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:46:53.338382169 (513h17m53.209465224s) [seg_pos={0 0}, upload=450410]: Begin upload segment...
Dec 27 00:46:53.338449107 (513h17m53.209532142s) [seg_pos={0 0}, upload=450410]: Done begin upload segment.
Dec 27 00:46:53.338502960 (513h17m53.209586005s) [noise=true , address=45.91.116.6:4973 , piece_id=IN4XC2IM7CMYJOUYOOUQHGPVZFNPQRKJ7RGVAJEWXA43RFKBSZZQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.338571059 (513h17m53.209654104s) [noise=true , address=77.90.31.16:4735 , piece_id=G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339312621 (513h17m53.210395656s) [noise=true , address=145.79.227.48:4302 , piece_id=FIQELWLIL3WTQ3GCZPYXRWZX6CJC5JJ2ERPMB4ACWEXRTCTKFBKQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339380901 (513h17m53.210463946s) [noise=true , address=77.90.31.11:4555 , piece_id=K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339792144 (513h17m53.210875189s) [noise=true , address=145.79.227.13:4804 , piece_id=G43OXV7NHD6HLEDOX4PDUEZSVKEL26PEVQ67HRRTJAA3DFIB6G3A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340132612 (513h17m53.211215647s) [noise=true , address=145.79.227.42:4121 , piece_id=CPPL66BYG3N3CVKR423TWTQTAGPHBSOJHNUWV44O2VD3NAQ4BFCQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340504771 (513h17m53.211587806s) [noise=true , address=145.79.227.50:4375 , piece_id=BCOBU74QR2CAJDGTHJRLXHGVYTLNEE27XB2ASUOMCQE3CYB5GASQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340841783 (513h17m53.211924828s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.341158887 (513h17m53.212241922s) [noise=true , address=145.79.227.52:4445 , piece_id=FKJJIBP4GUIEDRCNVZ6D5NHVMLGEZEYLBHLLVGREFNOBUPIUFIZA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.341439952 (513h17m53.212522997s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.342547922 (513h17m53.213630967s) [noise=true , address=145.79.227.16:4911 , piece_id=EMBD4H7WYYQP4OF5S7XI6RVZSPB2E2ZUJFRJYL55MOJMIHSXV2VA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343041732 (513h17m53.214124777s) [noise=true , address=145.79.227.3:4149 , piece_id=YBMTYYTTCUSD7OYCT7UYBD5DJ45OPJFUUQL7LZCZUTFTGEIEXRPQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343522357 (513h17m53.214605402s) [noise=true , address=145.79.227.51:4409 , piece_id=B6ILY6YECF6AKPPTK25MXH3V42UPW5PT26BRSEOGR5ZDCKCQWVYQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343608741 (513h17m53.214691786s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343960280 (513h17m53.215043316s) [noise=true , address=145.79.227.41:4086 , piece_id=HQZRCBSE3KVF7WXODTR7TQLC6DLFO7Z7P442KYGEEFKEQEDD24UA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.344310978 (513h17m53.215394023s) [noise=true , address=145.79.227.15:4875 , piece_id=H6FVGFPL52MLHHLMF4GAEH5IWLZAP7UO25YZXYSIVK4NUDYL3WOA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:55.045381986 (513h17m54.916465031s) [upload=450410]: Next returned nil segment
Dec 27 00:46:55.045399729 (513h17m54.916482794s) [upload=450410]: Waiting for error group managing segments...
Dec 27 00:46:55.335282585 (513h17m55.20636562s) [noise=true , address=145.79.227.50:4375 , piece_id=BCOBU74QR2CAJDGTHJRLXHGVYTLNEE27XB2ASUOMCQE3CYB5GASQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.338011361 (513h17m55.209094396s) [noise=true , address=145.79.227.3:4149 , piece_id=YBMTYYTTCUSD7OYCT7UYBD5DJ45OPJFUUQL7LZCZUTFTGEIEXRPQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.339463746 (513h17m55.210546791s) [noise=true , address=145.79.227.48:4302 , piece_id=FIQELWLIL3WTQ3GCZPYXRWZX6CJC5JJ2ERPMB4ACWEXRTCTKFBKQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.340722183 (513h17m55.211805228s) [noise=true , address=145.79.227.15:4875 , piece_id=H6FVGFPL52MLHHLMF4GAEH5IWLZAP7UO25YZXYSIVK4NUDYL3WOA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.343886809 (513h17m55.214969844s) [noise=true , address=145.79.227.51:4409 , piece_id=B6ILY6YECF6AKPPTK25MXH3V42UPW5PT26BRSEOGR5ZDCKCQWVYQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.349629133 (513h17m55.220712168s) [noise=true , address=145.79.227.52:4445 , piece_id=FKJJIBP4GUIEDRCNVZ6D5NHVMLGEZEYLBHLLVGREFNOBUPIUFIZA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.353120751 (513h17m55.224203786s) [noise=true , address=145.79.227.13:4804 , piece_id=G43OXV7NHD6HLEDOX4PDUEZSVKEL26PEVQ67HRRTJAA3DFIB6G3A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.357296662 (513h17m55.228379707s) [noise=true , address=145.79.227.16:4911 , piece_id=EMBD4H7WYYQP4OF5S7XI6RVZSPB2E2ZUJFRJYL55MOJMIHSXV2VA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.368751163 (513h17m55.239834208s) [noise=true , address=145.79.227.42:4121 , piece_id=CPPL66BYG3N3CVKR423TWTQTAGPHBSOJHNUWV44O2VD3NAQ4BFCQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.369848654 (513h17m55.240931689s) [noise=true , address=145.79.227.41:4086 , piece_id=HQZRCBSE3KVF7WXODTR7TQLC6DLFO7Z7P442KYGEEFKEQEDD24UA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.503905865 (513h17m55.3749889s) [noise=true , address=77.90.31.11:4555 , piece_id=K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.544674503 (513h17m55.415757548s) [noise=true , address=77.90.31.16:4735 , piece_id=G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.583254845 (513h17m55.45433788s) [noise=true , address=45.91.116.6:4973 , piece_id=IN4XC2IM7CMYJOUYOOUQHGPVZFNPQRKJ7RGVAJEWXA43RFKBSZZQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.583272228 (513h17m55.454355263s) [seg_pos={0 0}, upload=450410]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:46:55.583448484 (513h17m55.454531519s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection
Dec 27 00:46:55.583454526 (513h17m55.454537561s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583729449 (513h17m55.454812504s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 27 00:46:55.583748826 (513h17m55.454831881s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583765217 (513h17m55.454848252s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:46:55.583774265 (513h17m55.4548573s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583988202 (513h17m55.455071247s) [seg_pos={0 0}, upload=450410]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:46:55.584230393 (513h17m55.455313428s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:46:55.584253126 (513h17m55.455336171s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:46:55.883178679 (513h17m55.754261714s) [upload=450410]: Done uploading segments...
Dec 27 00:46:55.883199319 (513h17m55.754282354s) [upload=450410]: Upload finished. err: <nil>
Duration: 21.648831079s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [.....X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X.] duration:
6.103859823s trace:
https://jaeger.dev.storj.io/trace/613b4980e2af9b90
FAIL (2025-12-27 00:48:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A) context deadline exceeded
======LOGS======
ec 27 00:46:53.091927718 (513h17m52.963010763s) [upload=450410]: Starting upload
Dec 27 00:46:53.092014754 (513h17m52.963097789s) [upload=450410]: Done starting upload
Dec 27 00:46:53.092086631 (513h17m52.963169676s) [upload=450410]: Uploading segments...
Dec 27 00:46:53.092246345 (513h17m52.96332939s) [upload=450410]: Got next segment. Inline: false
Dec 27 00:46:53.092289137 (513h17m52.963372182s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:46:53.092294687 (513h17m52.963377732s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:46:53.338382169 (513h17m53.209465224s) [seg_pos={0 0}, upload=450410]: Begin upload segment...
Dec 27 00:46:53.338449107 (513h17m53.209532142s) [seg_pos={0 0}, upload=450410]: Done begin upload segment.
Dec 27 00:46:53.338502960 (513h17m53.209586005s) [noise=true , address=45.91.116.6:4973 , piece_id=IN4XC2IM7CMYJOUYOOUQHGPVZFNPQRKJ7RGVAJEWXA43RFKBSZZQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.338571059 (513h17m53.209654104s) [noise=true , address=77.90.31.16:4735 , piece_id=G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339312621 (513h17m53.210395656s) [noise=true , address=145.79.227.48:4302 , piece_id=FIQELWLIL3WTQ3GCZPYXRWZX6CJC5JJ2ERPMB4ACWEXRTCTKFBKQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339380901 (513h17m53.210463946s) [noise=true , address=77.90.31.11:4555 , piece_id=K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339792144 (513h17m53.210875189s) [noise=true , address=145.79.227.13:4804 , piece_id=G43OXV7NHD6HLEDOX4PDUEZSVKEL26PEVQ67HRRTJAA3DFIB6G3A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340132612 (513h17m53.211215647s) [noise=true , address=145.79.227.42:4121 , piece_id=CPPL66BYG3N3CVKR423TWTQTAGPHBSOJHNUWV44O2VD3NAQ4BFCQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340504771 (513h17m53.211587806s) [noise=true , address=145.79.227.50:4375 , piece_id=BCOBU74QR2CAJDGTHJRLXHGVYTLNEE27XB2ASUOMCQE3CYB5GASQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340841783 (513h17m53.211924828s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.341158887 (513h17m53.212241922s) [noise=true , address=145.79.227.52:4445 , piece_id=FKJJIBP4GUIEDRCNVZ6D5NHVMLGEZEYLBHLLVGREFNOBUPIUFIZA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.341439952 (513h17m53.212522997s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.342547922 (513h17m53.213630967s) [noise=true , address=145.79.227.16:4911 , piece_id=EMBD4H7WYYQP4OF5S7XI6RVZSPB2E2ZUJFRJYL55MOJMIHSXV2VA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343041732 (513h17m53.214124777s) [noise=true , address=145.79.227.3:4149 , piece_id=YBMTYYTTCUSD7OYCT7UYBD5DJ45OPJFUUQL7LZCZUTFTGEIEXRPQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343522357 (513h17m53.214605402s) [noise=true , address=145.79.227.51:4409 , piece_id=B6ILY6YECF6AKPPTK25MXH3V42UPW5PT26BRSEOGR5ZDCKCQWVYQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343608741 (513h17m53.214691786s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343960280 (513h17m53.215043316s) [noise=true , address=145.79.227.41:4086 , piece_id=HQZRCBSE3KVF7WXODTR7TQLC6DLFO7Z7P442KYGEEFKEQEDD24UA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.344310978 (513h17m53.215394023s) [noise=true , address=145.79.227.15:4875 , piece_id=H6FVGFPL52MLHHLMF4GAEH5IWLZAP7UO25YZXYSIVK4NUDYL3WOA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:55.045381986 (513h17m54.916465031s) [upload=450410]: Next returned nil segment
Dec 27 00:46:55.045399729 (513h17m54.916482794s) [upload=450410]: Waiting for error group managing segments...
Dec 27 00:46:55.335282585 (513h17m55.20636562s) [noise=true , address=145.79.227.50:4375 , piece_id=BCOBU74QR2CAJDGTHJRLXHGVYTLNEE27XB2ASUOMCQE3CYB5GASQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.338011361 (513h17m55.209094396s) [noise=true , address=145.79.227.3:4149 , piece_id=YBMTYYTTCUSD7OYCT7UYBD5DJ45OPJFUUQL7LZCZUTFTGEIEXRPQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.339463746 (513h17m55.210546791s) [noise=true , address=145.79.227.48:4302 , piece_id=FIQELWLIL3WTQ3GCZPYXRWZX6CJC5JJ2ERPMB4ACWEXRTCTKFBKQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.340722183 (513h17m55.211805228s) [noise=true , address=145.79.227.15:4875 , piece_id=H6FVGFPL52MLHHLMF4GAEH5IWLZAP7UO25YZXYSIVK4NUDYL3WOA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.343886809 (513h17m55.214969844s) [noise=true , address=145.79.227.51:4409 , piece_id=B6ILY6YECF6AKPPTK25MXH3V42UPW5PT26BRSEOGR5ZDCKCQWVYQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.349629133 (513h17m55.220712168s) [noise=true , address=145.79.227.52:4445 , piece_id=FKJJIBP4GUIEDRCNVZ6D5NHVMLGEZEYLBHLLVGREFNOBUPIUFIZA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.353120751 (513h17m55.224203786s) [noise=true , address=145.79.227.13:4804 , piece_id=G43OXV7NHD6HLEDOX4PDUEZSVKEL26PEVQ67HRRTJAA3DFIB6G3A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.357296662 (513h17m55.228379707s) [noise=true , address=145.79.227.16:4911 , piece_id=EMBD4H7WYYQP4OF5S7XI6RVZSPB2E2ZUJFRJYL55MOJMIHSXV2VA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.368751163 (513h17m55.239834208s) [noise=true , address=145.79.227.42:4121 , piece_id=CPPL66BYG3N3CVKR423TWTQTAGPHBSOJHNUWV44O2VD3NAQ4BFCQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.369848654 (513h17m55.240931689s) [noise=true , address=145.79.227.41:4086 , piece_id=HQZRCBSE3KVF7WXODTR7TQLC6DLFO7Z7P442KYGEEFKEQEDD24UA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.503905865 (513h17m55.3749889s) [noise=true , address=77.90.31.11:4555 , piece_id=K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.544674503 (513h17m55.415757548s) [noise=true , address=77.90.31.16:4735 , piece_id=G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.583254845 (513h17m55.45433788s) [noise=true , address=45.91.116.6:4973 , piece_id=IN4XC2IM7CMYJOUYOOUQHGPVZFNPQRKJ7RGVAJEWXA43RFKBSZZQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.583272228 (513h17m55.454355263s) [seg_pos={0 0}, upload=450410]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:46:55.583448484 (513h17m55.454531519s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection
Dec 27 00:46:55.583454526 (513h17m55.454537561s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583729449 (513h17m55.454812504s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 27 00:46:55.583748826 (513h17m55.454831881s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583765217 (513h17m55.454848252s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:46:55.583774265 (513h17m55.4548573s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583988202 (513h17m55.455071247s) [seg_pos={0 0}, upload=450410]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:46:55.584230393 (513h17m55.455313428s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:46:55.584253126 (513h17m55.455336171s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:46:55.883178679 (513h17m55.754261714s) [upload=450410]: Done uploading segments...
Dec 27 00:46:55.883199319 (513h17m55.754282354s) [upload=450410]: Upload finished. err: <nil>
Duration: 6.103889471s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
26.713172107s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:47:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 26.713201651s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [......X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....X] duration:
30.002731033s trace:
https://jaeger.dev.storj.io/trace/613b4980e2af9b90
FAIL (2025-12-27 00:47:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A) context deadline exceeded
======LOGS======
ec 27 00:46:53.091927718 (513h17m52.963010763s) [upload=450410]: Starting upload
Dec 27 00:46:53.092014754 (513h17m52.963097789s) [upload=450410]: Done starting upload
Dec 27 00:46:53.092086631 (513h17m52.963169676s) [upload=450410]: Uploading segments...
Dec 27 00:46:53.092246345 (513h17m52.96332939s) [upload=450410]: Got next segment. Inline: false
Dec 27 00:46:53.092289137 (513h17m52.963372182s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:46:53.092294687 (513h17m52.963377732s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:46:53.338382169 (513h17m53.209465224s) [seg_pos={0 0}, upload=450410]: Begin upload segment...
Dec 27 00:46:53.338449107 (513h17m53.209532142s) [seg_pos={0 0}, upload=450410]: Done begin upload segment.
Dec 27 00:46:53.338502960 (513h17m53.209586005s) [noise=true , address=45.91.116.6:4973 , piece_id=IN4XC2IM7CMYJOUYOOUQHGPVZFNPQRKJ7RGVAJEWXA43RFKBSZZQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.338571059 (513h17m53.209654104s) [noise=true , address=77.90.31.16:4735 , piece_id=G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339312621 (513h17m53.210395656s) [noise=true , address=145.79.227.48:4302 , piece_id=FIQELWLIL3WTQ3GCZPYXRWZX6CJC5JJ2ERPMB4ACWEXRTCTKFBKQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339380901 (513h17m53.210463946s) [noise=true , address=77.90.31.11:4555 , piece_id=K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.339792144 (513h17m53.210875189s) [noise=true , address=145.79.227.13:4804 , piece_id=G43OXV7NHD6HLEDOX4PDUEZSVKEL26PEVQ67HRRTJAA3DFIB6G3A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340132612 (513h17m53.211215647s) [noise=true , address=145.79.227.42:4121 , piece_id=CPPL66BYG3N3CVKR423TWTQTAGPHBSOJHNUWV44O2VD3NAQ4BFCQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340504771 (513h17m53.211587806s) [noise=true , address=145.79.227.50:4375 , piece_id=BCOBU74QR2CAJDGTHJRLXHGVYTLNEE27XB2ASUOMCQE3CYB5GASQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.340841783 (513h17m53.211924828s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.341158887 (513h17m53.212241922s) [noise=true , address=145.79.227.52:4445 , piece_id=FKJJIBP4GUIEDRCNVZ6D5NHVMLGEZEYLBHLLVGREFNOBUPIUFIZA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.341439952 (513h17m53.212522997s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.342547922 (513h17m53.213630967s) [noise=true , address=145.79.227.16:4911 , piece_id=EMBD4H7WYYQP4OF5S7XI6RVZSPB2E2ZUJFRJYL55MOJMIHSXV2VA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343041732 (513h17m53.214124777s) [noise=true , address=145.79.227.3:4149 , piece_id=YBMTYYTTCUSD7OYCT7UYBD5DJ45OPJFUUQL7LZCZUTFTGEIEXRPQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343522357 (513h17m53.214605402s) [noise=true , address=145.79.227.51:4409 , piece_id=B6ILY6YECF6AKPPTK25MXH3V42UPW5PT26BRSEOGR5ZDCKCQWVYQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343608741 (513h17m53.214691786s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.343960280 (513h17m53.215043316s) [noise=true , address=145.79.227.41:4086 , piece_id=HQZRCBSE3KVF7WXODTR7TQLC6DLFO7Z7P442KYGEEFKEQEDD24UA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:53.344310978 (513h17m53.215394023s) [noise=true , address=145.79.227.15:4875 , piece_id=H6FVGFPL52MLHHLMF4GAEH5IWLZAP7UO25YZXYSIVK4NUDYL3WOA, seg_pos={0 0}, upload=450410]: Uploading piece...
Dec 27 00:46:55.045381986 (513h17m54.916465031s) [upload=450410]: Next returned nil segment
Dec 27 00:46:55.045399729 (513h17m54.916482794s) [upload=450410]: Waiting for error group managing segments...
Dec 27 00:46:55.335282585 (513h17m55.20636562s) [noise=true , address=145.79.227.50:4375 , piece_id=BCOBU74QR2CAJDGTHJRLXHGVYTLNEE27XB2ASUOMCQE3CYB5GASQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.338011361 (513h17m55.209094396s) [noise=true , address=145.79.227.3:4149 , piece_id=YBMTYYTTCUSD7OYCT7UYBD5DJ45OPJFUUQL7LZCZUTFTGEIEXRPQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.339463746 (513h17m55.210546791s) [noise=true , address=145.79.227.48:4302 , piece_id=FIQELWLIL3WTQ3GCZPYXRWZX6CJC5JJ2ERPMB4ACWEXRTCTKFBKQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.340722183 (513h17m55.211805228s) [noise=true , address=145.79.227.15:4875 , piece_id=H6FVGFPL52MLHHLMF4GAEH5IWLZAP7UO25YZXYSIVK4NUDYL3WOA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.343886809 (513h17m55.214969844s) [noise=true , address=145.79.227.51:4409 , piece_id=B6ILY6YECF6AKPPTK25MXH3V42UPW5PT26BRSEOGR5ZDCKCQWVYQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.349629133 (513h17m55.220712168s) [noise=true , address=145.79.227.52:4445 , piece_id=FKJJIBP4GUIEDRCNVZ6D5NHVMLGEZEYLBHLLVGREFNOBUPIUFIZA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.353120751 (513h17m55.224203786s) [noise=true , address=145.79.227.13:4804 , piece_id=G43OXV7NHD6HLEDOX4PDUEZSVKEL26PEVQ67HRRTJAA3DFIB6G3A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.357296662 (513h17m55.228379707s) [noise=true , address=145.79.227.16:4911 , piece_id=EMBD4H7WYYQP4OF5S7XI6RVZSPB2E2ZUJFRJYL55MOJMIHSXV2VA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.368751163 (513h17m55.239834208s) [noise=true , address=145.79.227.42:4121 , piece_id=CPPL66BYG3N3CVKR423TWTQTAGPHBSOJHNUWV44O2VD3NAQ4BFCQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.369848654 (513h17m55.240931689s) [noise=true , address=145.79.227.41:4086 , piece_id=HQZRCBSE3KVF7WXODTR7TQLC6DLFO7Z7P442KYGEEFKEQEDD24UA, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.503905865 (513h17m55.3749889s) [noise=true , address=77.90.31.11:4555 , piece_id=K45O6U32IACTY7XNF455MM7CALX3PRLWNLX7AZMKACOXB6IO4O7A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.544674503 (513h17m55.415757548s) [noise=true , address=77.90.31.16:4735 , piece_id=G2JGPCUVMLKOAFDBNTEZSFOZMCPPTZOGJAWWMNPVAVDVASJRRM6A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.583254845 (513h17m55.45433788s) [noise=true , address=45.91.116.6:4973 , piece_id=IN4XC2IM7CMYJOUYOOUQHGPVZFNPQRKJ7RGVAJEWXA43RFKBSZZQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: <nil>
Dec 27 00:46:55.583272228 (513h17m55.454355263s) [seg_pos={0 0}, upload=450410]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:46:55.583448484 (513h17m55.454531519s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60522->45.91.116.4:4939: use of closed network connection
Dec 27 00:46:55.583454526 (513h17m55.454537561s) [noise=true , address=45.91.116.4:4939 , piece_id=UWNLUGMNSGX5B2Q4ILMTEUULYNYMFUR5DL3BVIHA72TTAKIDEZPQ, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583729449 (513h17m55.454812504s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 27 00:46:55.583748826 (513h17m55.454831881s) [noise=true , address=45.91.116.12:4225 , piece_id=BRCBPMTLD2E2QKP66X6FASRC2TWDEP5ZXFJEPQ5OYGIXLBAE6N6A, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583765217 (513h17m55.454848252s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:46:55.583774265 (513h17m55.4548573s) [noise=true , address=45.91.116.14:4191 , piece_id=5JVE5AR72EAWYRHAVFZXPNVMJKEZD3OBL6WDGJIPPW6WVXA5K44Q, seg_pos={0 0}, upload=450410]: Long tail cancellation due to optimal threshold
Dec 27 00:46:55.583988202 (513h17m55.455071247s) [seg_pos={0 0}, upload=450410]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:46:55.584230393 (513h17m55.455313428s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:46:55.584253126 (513h17m55.455336171s) [upload=450410]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:46:55.883178679 (513h17m55.754261714s) [upload=450410]: Done uploading segments...
Dec 27 00:46:55.883199319 (513h17m55.754282354s) [upload=450410]: Upload finished. err: <nil>
Duration: 30.002754238s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
30.221505455s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:46:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 30.221524722s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
36.20572918s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:46:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 36.205758326s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.......X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX....] duration:
5.917544906s trace:
https://jaeger.dev.storj.io/trace/533cafd369c9ef95
FAIL (2025-12-27 00:45:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q) context deadline exceeded
======LOGS======
357]: Starting upload
Dec 27 00:43:13.647580027 (513h14m13.518663072s) [upload=450357]: Done starting upload
Dec 27 00:43:13.647712699 (513h14m13.518795734s) [upload=450357]: Uploading segments...
Dec 27 00:43:13.647829061 (513h14m13.518912096s) [upload=450357]: Got next segment. Inline: false
Dec 27 00:43:13.647886079 (513h14m13.518969124s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:43:13.647892362 (513h14m13.518975407s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:43:13.896495033 (513h14m13.767578078s) [seg_pos={0 0}, upload=450357]: Begin upload segment...
Dec 27 00:43:13.896558864 (513h14m13.767641899s) [seg_pos={0 0}, upload=450357]: Done begin upload segment.
Dec 27 00:43:13.896599492 (513h14m13.767682527s) [noise=true , address=145.79.227.36:4519 , piece_id=LUWN3PRDZQGW65O3K5SG4FIB5JBERIW5QWKYKYUTJUAQTMHABBOA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.896738566 (513h14m13.767821611s) [noise=true , address=45.91.116.13:4156 , piece_id=PEGHBWEQXJMYAHVVKD4F2EHFLUSJEIUTK4KXWB7ONGMBRU6VHDRA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897306407 (513h14m13.768389452s) [noise=true , address=45.91.116.5:4298 , piece_id=565NHL6TPU5DDO2VHLHZPU4MF3O244WYEBWRAFEHJEXPU4ZGJ7XA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897602832 (513h14m13.768685877s) [noise=true , address=145.79.227.50:4375 , piece_id=IEPVFXIPQCUNQLO2A2NR4TOVRYRVBNFYVOSP33TAADNCL2XLHKEQ, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897926558 (513h14m13.769009593s) [noise=true , address=145.79.227.38:4978 , piece_id=ECI3U6UBNEKMNACECKBAQLK423S63YGUBIHGELZ75E7YN3UAMWVA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898244684 (513h14m13.769327719s) [noise=true , address=145.79.227.45:4229 , piece_id=REFUVKI7DB3U464KOM3HP66SOHBEUD5C7QO4PHHFKWRIU3VOTFDA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898517794 (513h14m13.769600839s) [noise=true , address=77.90.31.19:4842 , piece_id=ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898801805 (513h14m13.76988484s) [noise=true , address=145.79.227.2:4115 , piece_id=3ANBT2RYXVTSEHSPOYR3LMUTCGYZR7WYYNVVJ2YQKHN6BEM5QUTA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.899116153 (513h14m13.770199188s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.900571314 (513h14m13.771654349s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.900724496 (513h14m13.771807531s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.902054269 (513h14m13.773137304s) [noise=true , address=45.91.116.9:4263 , piece_id=LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.903487227 (513h14m13.774570523s) [noise=true , address=145.79.227.47:4267 , piece_id=4RHTBMFLM3Q2DAJXG7CKQ3XGPOT2N7FH4I6NMWQ7K2C6HJG34ARA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.904408811 (513h14m13.775491847s) [noise=true , address=45.91.116.8:4046 , piece_id=IKWDIRO5HBIJ3XBNJQDI4VL7H3UG7BZPMTMXRUEMOEOUTDKOHWGA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.907273937 (513h14m13.778356972s) [noise=true , address=145.79.227.49:4338 , piece_id=HYQYAJAXR7QSD45JR35GBQEIP6LGKID2N2V5TX6CYFUDYKXNCB7A, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.910120928 (513h14m13.781203963s) [noise=true , address=45.91.116.11:4119 , piece_id=QE3QOGTULK3F2JQCRMI6Y5ZDX5RZPKJGMVWUAGVO2GQN7FVQRP5Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:15.597774489 (513h14m15.468857524s) [upload=450357]: Next returned nil segment
Dec 27 00:43:15.597788597 (513h14m15.468871632s) [upload=450357]: Waiting for error group managing segments...
Dec 27 00:43:15.892620876 (513h14m15.763703911s) [noise=true , address=145.79.227.50:4375 , piece_id=IEPVFXIPQCUNQLO2A2NR4TOVRYRVBNFYVOSP33TAADNCL2XLHKEQ, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.900629124 (513h14m15.771712169s) [noise=true , address=145.79.227.45:4229 , piece_id=REFUVKI7DB3U464KOM3HP66SOHBEUD5C7QO4PHHFKWRIU3VOTFDA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.905117851 (513h14m15.776200896s) [noise=true , address=145.79.227.2:4115 , piece_id=3ANBT2RYXVTSEHSPOYR3LMUTCGYZR7WYYNVVJ2YQKHN6BEM5QUTA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.905595411 (513h14m15.776678456s) [noise=true , address=145.79.227.38:4978 , piece_id=ECI3U6UBNEKMNACECKBAQLK423S63YGUBIHGELZ75E7YN3UAMWVA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.906622075 (513h14m15.77770511s) [noise=true , address=145.79.227.36:4519 , piece_id=LUWN3PRDZQGW65O3K5SG4FIB5JBERIW5QWKYKYUTJUAQTMHABBOA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.923866802 (513h14m15.794949837s) [noise=true , address=145.79.227.47:4267 , piece_id=4RHTBMFLM3Q2DAJXG7CKQ3XGPOT2N7FH4I6NMWQ7K2C6HJG34ARA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.976109839 (513h14m15.847192884s) [noise=true , address=145.79.227.49:4338 , piece_id=HYQYAJAXR7QSD45JR35GBQEIP6LGKID2N2V5TX6CYFUDYKXNCB7A, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.068723638 (513h14m15.939806673s) [noise=true , address=77.90.31.19:4842 , piece_id=ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.256785211 (513h14m16.127868246s) [noise=true , address=45.91.116.13:4156 , piece_id=PEGHBWEQXJMYAHVVKD4F2EHFLUSJEIUTK4KXWB7ONGMBRU6VHDRA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.261236718 (513h14m16.132319753s) [noise=true , address=45.91.116.11:4119 , piece_id=QE3QOGTULK3F2JQCRMI6Y5ZDX5RZPKJGMVWUAGVO2GQN7FVQRP5Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.264852523 (513h14m16.135935568s) [noise=true , address=45.91.116.8:4046 , piece_id=IKWDIRO5HBIJ3XBNJQDI4VL7H3UG7BZPMTMXRUEMOEOUTDKOHWGA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.297360141 (513h14m16.168443196s) [noise=true , address=45.91.116.5:4298 , piece_id=565NHL6TPU5DDO2VHLHZPU4MF3O244WYEBWRAFEHJEXPU4ZGJ7XA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.373538565 (513h14m16.2446216s) [noise=true , address=45.91.116.9:4263 , piece_id=LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.373563493 (513h14m16.244646578s) [seg_pos={0 0}, upload=450357]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:43:16.373829158 (513h14m16.244912193s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:43:16.373837363 (513h14m16.244920398s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.373890735 (513h14m16.24497382s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection
Dec 27 00:43:16.373900754 (513h14m16.244983789s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.374121756 (513h14m16.245204791s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:55566->45.91.116.4:4937: use of closed network connection; context canceled
Dec 27 00:43:16.374128659 (513h14m16.245211704s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.374142144 (513h14m16.245225189s) [seg_pos={0 0}, upload=450357]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:43:16.374271821 (513h14m16.245354856s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:43:16.374277532 (513h14m16.245360597s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:43:16.671060053 (513h14m16.542143108s) [upload=450357]: Done uploading segments...
Dec 27 00:43:16.671085672 (513h14m16.542168717s) [upload=450357]: Upload finished. err: <nil>
Duration: 5.91757292s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
9.230717458s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:45:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 9.230750821s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [........X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX...] duration:
6.252398591s trace:
https://jaeger.dev.storj.io/trace/533cafd369c9ef95
FAIL (2025-12-27 00:45:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q) context deadline exceeded
======LOGS======
357]: Starting upload
Dec 27 00:43:13.647580027 (513h14m13.518663072s) [upload=450357]: Done starting upload
Dec 27 00:43:13.647712699 (513h14m13.518795734s) [upload=450357]: Uploading segments...
Dec 27 00:43:13.647829061 (513h14m13.518912096s) [upload=450357]: Got next segment. Inline: false
Dec 27 00:43:13.647886079 (513h14m13.518969124s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:43:13.647892362 (513h14m13.518975407s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:43:13.896495033 (513h14m13.767578078s) [seg_pos={0 0}, upload=450357]: Begin upload segment...
Dec 27 00:43:13.896558864 (513h14m13.767641899s) [seg_pos={0 0}, upload=450357]: Done begin upload segment.
Dec 27 00:43:13.896599492 (513h14m13.767682527s) [noise=true , address=145.79.227.36:4519 , piece_id=LUWN3PRDZQGW65O3K5SG4FIB5JBERIW5QWKYKYUTJUAQTMHABBOA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.896738566 (513h14m13.767821611s) [noise=true , address=45.91.116.13:4156 , piece_id=PEGHBWEQXJMYAHVVKD4F2EHFLUSJEIUTK4KXWB7ONGMBRU6VHDRA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897306407 (513h14m13.768389452s) [noise=true , address=45.91.116.5:4298 , piece_id=565NHL6TPU5DDO2VHLHZPU4MF3O244WYEBWRAFEHJEXPU4ZGJ7XA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897602832 (513h14m13.768685877s) [noise=true , address=145.79.227.50:4375 , piece_id=IEPVFXIPQCUNQLO2A2NR4TOVRYRVBNFYVOSP33TAADNCL2XLHKEQ, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897926558 (513h14m13.769009593s) [noise=true , address=145.79.227.38:4978 , piece_id=ECI3U6UBNEKMNACECKBAQLK423S63YGUBIHGELZ75E7YN3UAMWVA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898244684 (513h14m13.769327719s) [noise=true , address=145.79.227.45:4229 , piece_id=REFUVKI7DB3U464KOM3HP66SOHBEUD5C7QO4PHHFKWRIU3VOTFDA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898517794 (513h14m13.769600839s) [noise=true , address=77.90.31.19:4842 , piece_id=ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898801805 (513h14m13.76988484s) [noise=true , address=145.79.227.2:4115 , piece_id=3ANBT2RYXVTSEHSPOYR3LMUTCGYZR7WYYNVVJ2YQKHN6BEM5QUTA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.899116153 (513h14m13.770199188s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.900571314 (513h14m13.771654349s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.900724496 (513h14m13.771807531s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.902054269 (513h14m13.773137304s) [noise=true , address=45.91.116.9:4263 , piece_id=LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.903487227 (513h14m13.774570523s) [noise=true , address=145.79.227.47:4267 , piece_id=4RHTBMFLM3Q2DAJXG7CKQ3XGPOT2N7FH4I6NMWQ7K2C6HJG34ARA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.904408811 (513h14m13.775491847s) [noise=true , address=45.91.116.8:4046 , piece_id=IKWDIRO5HBIJ3XBNJQDI4VL7H3UG7BZPMTMXRUEMOEOUTDKOHWGA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.907273937 (513h14m13.778356972s) [noise=true , address=145.79.227.49:4338 , piece_id=HYQYAJAXR7QSD45JR35GBQEIP6LGKID2N2V5TX6CYFUDYKXNCB7A, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.910120928 (513h14m13.781203963s) [noise=true , address=45.91.116.11:4119 , piece_id=QE3QOGTULK3F2JQCRMI6Y5ZDX5RZPKJGMVWUAGVO2GQN7FVQRP5Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:15.597774489 (513h14m15.468857524s) [upload=450357]: Next returned nil segment
Dec 27 00:43:15.597788597 (513h14m15.468871632s) [upload=450357]: Waiting for error group managing segments...
Dec 27 00:43:15.892620876 (513h14m15.763703911s) [noise=true , address=145.79.227.50:4375 , piece_id=IEPVFXIPQCUNQLO2A2NR4TOVRYRVBNFYVOSP33TAADNCL2XLHKEQ, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.900629124 (513h14m15.771712169s) [noise=true , address=145.79.227.45:4229 , piece_id=REFUVKI7DB3U464KOM3HP66SOHBEUD5C7QO4PHHFKWRIU3VOTFDA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.905117851 (513h14m15.776200896s) [noise=true , address=145.79.227.2:4115 , piece_id=3ANBT2RYXVTSEHSPOYR3LMUTCGYZR7WYYNVVJ2YQKHN6BEM5QUTA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.905595411 (513h14m15.776678456s) [noise=true , address=145.79.227.38:4978 , piece_id=ECI3U6UBNEKMNACECKBAQLK423S63YGUBIHGELZ75E7YN3UAMWVA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.906622075 (513h14m15.77770511s) [noise=true , address=145.79.227.36:4519 , piece_id=LUWN3PRDZQGW65O3K5SG4FIB5JBERIW5QWKYKYUTJUAQTMHABBOA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.923866802 (513h14m15.794949837s) [noise=true , address=145.79.227.47:4267 , piece_id=4RHTBMFLM3Q2DAJXG7CKQ3XGPOT2N7FH4I6NMWQ7K2C6HJG34ARA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.976109839 (513h14m15.847192884s) [noise=true , address=145.79.227.49:4338 , piece_id=HYQYAJAXR7QSD45JR35GBQEIP6LGKID2N2V5TX6CYFUDYKXNCB7A, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.068723638 (513h14m15.939806673s) [noise=true , address=77.90.31.19:4842 , piece_id=ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.256785211 (513h14m16.127868246s) [noise=true , address=45.91.116.13:4156 , piece_id=PEGHBWEQXJMYAHVVKD4F2EHFLUSJEIUTK4KXWB7ONGMBRU6VHDRA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.261236718 (513h14m16.132319753s) [noise=true , address=45.91.116.11:4119 , piece_id=QE3QOGTULK3F2JQCRMI6Y5ZDX5RZPKJGMVWUAGVO2GQN7FVQRP5Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.264852523 (513h14m16.135935568s) [noise=true , address=45.91.116.8:4046 , piece_id=IKWDIRO5HBIJ3XBNJQDI4VL7H3UG7BZPMTMXRUEMOEOUTDKOHWGA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.297360141 (513h14m16.168443196s) [noise=true , address=45.91.116.5:4298 , piece_id=565NHL6TPU5DDO2VHLHZPU4MF3O244WYEBWRAFEHJEXPU4ZGJ7XA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.373538565 (513h14m16.2446216s) [noise=true , address=45.91.116.9:4263 , piece_id=LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.373563493 (513h14m16.244646578s) [seg_pos={0 0}, upload=450357]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:43:16.373829158 (513h14m16.244912193s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:43:16.373837363 (513h14m16.244920398s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.373890735 (513h14m16.24497382s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection
Dec 27 00:43:16.373900754 (513h14m16.244983789s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.374121756 (513h14m16.245204791s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:55566->45.91.116.4:4937: use of closed network connection; context canceled
Dec 27 00:43:16.374128659 (513h14m16.245211704s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.374142144 (513h14m16.245225189s) [seg_pos={0 0}, upload=450357]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:43:16.374271821 (513h14m16.245354856s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:43:16.374277532 (513h14m16.245360597s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:43:16.671060053 (513h14m16.542143108s) [upload=450357]: Done uploading segments...
Dec 27 00:43:16.671085672 (513h14m16.542168717s) [upload=450357]: Upload finished. err: <nil>
Duration: 6.252425823s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
15.170545029s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:44:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 15.170574696s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.........X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX..] duration:
20.138015935s trace:
https://jaeger.dev.storj.io/trace/533cafd369c9ef95
FAIL (2025-12-27 00:44:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q) context deadline exceeded
======LOGS======
357]: Starting upload
Dec 27 00:43:13.647580027 (513h14m13.518663072s) [upload=450357]: Done starting upload
Dec 27 00:43:13.647712699 (513h14m13.518795734s) [upload=450357]: Uploading segments...
Dec 27 00:43:13.647829061 (513h14m13.518912096s) [upload=450357]: Got next segment. Inline: false
Dec 27 00:43:13.647886079 (513h14m13.518969124s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:43:13.647892362 (513h14m13.518975407s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:43:13.896495033 (513h14m13.767578078s) [seg_pos={0 0}, upload=450357]: Begin upload segment...
Dec 27 00:43:13.896558864 (513h14m13.767641899s) [seg_pos={0 0}, upload=450357]: Done begin upload segment.
Dec 27 00:43:13.896599492 (513h14m13.767682527s) [noise=true , address=145.79.227.36:4519 , piece_id=LUWN3PRDZQGW65O3K5SG4FIB5JBERIW5QWKYKYUTJUAQTMHABBOA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.896738566 (513h14m13.767821611s) [noise=true , address=45.91.116.13:4156 , piece_id=PEGHBWEQXJMYAHVVKD4F2EHFLUSJEIUTK4KXWB7ONGMBRU6VHDRA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897306407 (513h14m13.768389452s) [noise=true , address=45.91.116.5:4298 , piece_id=565NHL6TPU5DDO2VHLHZPU4MF3O244WYEBWRAFEHJEXPU4ZGJ7XA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897602832 (513h14m13.768685877s) [noise=true , address=145.79.227.50:4375 , piece_id=IEPVFXIPQCUNQLO2A2NR4TOVRYRVBNFYVOSP33TAADNCL2XLHKEQ, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897926558 (513h14m13.769009593s) [noise=true , address=145.79.227.38:4978 , piece_id=ECI3U6UBNEKMNACECKBAQLK423S63YGUBIHGELZ75E7YN3UAMWVA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898244684 (513h14m13.769327719s) [noise=true , address=145.79.227.45:4229 , piece_id=REFUVKI7DB3U464KOM3HP66SOHBEUD5C7QO4PHHFKWRIU3VOTFDA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898517794 (513h14m13.769600839s) [noise=true , address=77.90.31.19:4842 , piece_id=ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898801805 (513h14m13.76988484s) [noise=true , address=145.79.227.2:4115 , piece_id=3ANBT2RYXVTSEHSPOYR3LMUTCGYZR7WYYNVVJ2YQKHN6BEM5QUTA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.899116153 (513h14m13.770199188s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.900571314 (513h14m13.771654349s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.900724496 (513h14m13.771807531s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.902054269 (513h14m13.773137304s) [noise=true , address=45.91.116.9:4263 , piece_id=LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.903487227 (513h14m13.774570523s) [noise=true , address=145.79.227.47:4267 , piece_id=4RHTBMFLM3Q2DAJXG7CKQ3XGPOT2N7FH4I6NMWQ7K2C6HJG34ARA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.904408811 (513h14m13.775491847s) [noise=true , address=45.91.116.8:4046 , piece_id=IKWDIRO5HBIJ3XBNJQDI4VL7H3UG7BZPMTMXRUEMOEOUTDKOHWGA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.907273937 (513h14m13.778356972s) [noise=true , address=145.79.227.49:4338 , piece_id=HYQYAJAXR7QSD45JR35GBQEIP6LGKID2N2V5TX6CYFUDYKXNCB7A, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.910120928 (513h14m13.781203963s) [noise=true , address=45.91.116.11:4119 , piece_id=QE3QOGTULK3F2JQCRMI6Y5ZDX5RZPKJGMVWUAGVO2GQN7FVQRP5Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:15.597774489 (513h14m15.468857524s) [upload=450357]: Next returned nil segment
Dec 27 00:43:15.597788597 (513h14m15.468871632s) [upload=450357]: Waiting for error group managing segments...
Dec 27 00:43:15.892620876 (513h14m15.763703911s) [noise=true , address=145.79.227.50:4375 , piece_id=IEPVFXIPQCUNQLO2A2NR4TOVRYRVBNFYVOSP33TAADNCL2XLHKEQ, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.900629124 (513h14m15.771712169s) [noise=true , address=145.79.227.45:4229 , piece_id=REFUVKI7DB3U464KOM3HP66SOHBEUD5C7QO4PHHFKWRIU3VOTFDA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.905117851 (513h14m15.776200896s) [noise=true , address=145.79.227.2:4115 , piece_id=3ANBT2RYXVTSEHSPOYR3LMUTCGYZR7WYYNVVJ2YQKHN6BEM5QUTA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.905595411 (513h14m15.776678456s) [noise=true , address=145.79.227.38:4978 , piece_id=ECI3U6UBNEKMNACECKBAQLK423S63YGUBIHGELZ75E7YN3UAMWVA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.906622075 (513h14m15.77770511s) [noise=true , address=145.79.227.36:4519 , piece_id=LUWN3PRDZQGW65O3K5SG4FIB5JBERIW5QWKYKYUTJUAQTMHABBOA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.923866802 (513h14m15.794949837s) [noise=true , address=145.79.227.47:4267 , piece_id=4RHTBMFLM3Q2DAJXG7CKQ3XGPOT2N7FH4I6NMWQ7K2C6HJG34ARA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.976109839 (513h14m15.847192884s) [noise=true , address=145.79.227.49:4338 , piece_id=HYQYAJAXR7QSD45JR35GBQEIP6LGKID2N2V5TX6CYFUDYKXNCB7A, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.068723638 (513h14m15.939806673s) [noise=true , address=77.90.31.19:4842 , piece_id=ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.256785211 (513h14m16.127868246s) [noise=true , address=45.91.116.13:4156 , piece_id=PEGHBWEQXJMYAHVVKD4F2EHFLUSJEIUTK4KXWB7ONGMBRU6VHDRA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.261236718 (513h14m16.132319753s) [noise=true , address=45.91.116.11:4119 , piece_id=QE3QOGTULK3F2JQCRMI6Y5ZDX5RZPKJGMVWUAGVO2GQN7FVQRP5Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.264852523 (513h14m16.135935568s) [noise=true , address=45.91.116.8:4046 , piece_id=IKWDIRO5HBIJ3XBNJQDI4VL7H3UG7BZPMTMXRUEMOEOUTDKOHWGA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.297360141 (513h14m16.168443196s) [noise=true , address=45.91.116.5:4298 , piece_id=565NHL6TPU5DDO2VHLHZPU4MF3O244WYEBWRAFEHJEXPU4ZGJ7XA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.373538565 (513h14m16.2446216s) [noise=true , address=45.91.116.9:4263 , piece_id=LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.373563493 (513h14m16.244646578s) [seg_pos={0 0}, upload=450357]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:43:16.373829158 (513h14m16.244912193s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:43:16.373837363 (513h14m16.244920398s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.373890735 (513h14m16.24497382s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection
Dec 27 00:43:16.373900754 (513h14m16.244983789s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.374121756 (513h14m16.245204791s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:55566->45.91.116.4:4937: use of closed network connection; context canceled
Dec 27 00:43:16.374128659 (513h14m16.245211704s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.374142144 (513h14m16.245225189s) [seg_pos={0 0}, upload=450357]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:43:16.374271821 (513h14m16.245354856s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:43:16.374277532 (513h14m16.245360597s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:43:16.671060053 (513h14m16.542143108s) [upload=450357]: Done uploading segments...
Dec 27 00:43:16.671085672 (513h14m16.542168717s) [upload=450357]: Upload finished. err: <nil>
Duration: 20.138056292s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
35.620202143s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:44:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 35.620225218s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [X.........X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX.] duration:
5.675076209s trace:
https://jaeger.dev.storj.io/trace/533cafd369c9ef95
FAIL (2025-12-27 00:44:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q) context deadline exceeded
======LOGS======
357]: Starting upload
Dec 27 00:43:13.647580027 (513h14m13.518663072s) [upload=450357]: Done starting upload
Dec 27 00:43:13.647712699 (513h14m13.518795734s) [upload=450357]: Uploading segments...
Dec 27 00:43:13.647829061 (513h14m13.518912096s) [upload=450357]: Got next segment. Inline: false
Dec 27 00:43:13.647886079 (513h14m13.518969124s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:43:13.647892362 (513h14m13.518975407s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:43:13.896495033 (513h14m13.767578078s) [seg_pos={0 0}, upload=450357]: Begin upload segment...
Dec 27 00:43:13.896558864 (513h14m13.767641899s) [seg_pos={0 0}, upload=450357]: Done begin upload segment.
Dec 27 00:43:13.896599492 (513h14m13.767682527s) [noise=true , address=145.79.227.36:4519 , piece_id=LUWN3PRDZQGW65O3K5SG4FIB5JBERIW5QWKYKYUTJUAQTMHABBOA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.896738566 (513h14m13.767821611s) [noise=true , address=45.91.116.13:4156 , piece_id=PEGHBWEQXJMYAHVVKD4F2EHFLUSJEIUTK4KXWB7ONGMBRU6VHDRA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897306407 (513h14m13.768389452s) [noise=true , address=45.91.116.5:4298 , piece_id=565NHL6TPU5DDO2VHLHZPU4MF3O244WYEBWRAFEHJEXPU4ZGJ7XA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897602832 (513h14m13.768685877s) [noise=true , address=145.79.227.50:4375 , piece_id=IEPVFXIPQCUNQLO2A2NR4TOVRYRVBNFYVOSP33TAADNCL2XLHKEQ, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897926558 (513h14m13.769009593s) [noise=true , address=145.79.227.38:4978 , piece_id=ECI3U6UBNEKMNACECKBAQLK423S63YGUBIHGELZ75E7YN3UAMWVA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898244684 (513h14m13.769327719s) [noise=true , address=145.79.227.45:4229 , piece_id=REFUVKI7DB3U464KOM3HP66SOHBEUD5C7QO4PHHFKWRIU3VOTFDA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898517794 (513h14m13.769600839s) [noise=true , address=77.90.31.19:4842 , piece_id=ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898801805 (513h14m13.76988484s) [noise=true , address=145.79.227.2:4115 , piece_id=3ANBT2RYXVTSEHSPOYR3LMUTCGYZR7WYYNVVJ2YQKHN6BEM5QUTA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.899116153 (513h14m13.770199188s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.900571314 (513h14m13.771654349s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.900724496 (513h14m13.771807531s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.902054269 (513h14m13.773137304s) [noise=true , address=45.91.116.9:4263 , piece_id=LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.903487227 (513h14m13.774570523s) [noise=true , address=145.79.227.47:4267 , piece_id=4RHTBMFLM3Q2DAJXG7CKQ3XGPOT2N7FH4I6NMWQ7K2C6HJG34ARA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.904408811 (513h14m13.775491847s) [noise=true , address=45.91.116.8:4046 , piece_id=IKWDIRO5HBIJ3XBNJQDI4VL7H3UG7BZPMTMXRUEMOEOUTDKOHWGA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.907273937 (513h14m13.778356972s) [noise=true , address=145.79.227.49:4338 , piece_id=HYQYAJAXR7QSD45JR35GBQEIP6LGKID2N2V5TX6CYFUDYKXNCB7A, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.910120928 (513h14m13.781203963s) [noise=true , address=45.91.116.11:4119 , piece_id=QE3QOGTULK3F2JQCRMI6Y5ZDX5RZPKJGMVWUAGVO2GQN7FVQRP5Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:15.597774489 (513h14m15.468857524s) [upload=450357]: Next returned nil segment
Dec 27 00:43:15.597788597 (513h14m15.468871632s) [upload=450357]: Waiting for error group managing segments...
Dec 27 00:43:15.892620876 (513h14m15.763703911s) [noise=true , address=145.79.227.50:4375 , piece_id=IEPVFXIPQCUNQLO2A2NR4TOVRYRVBNFYVOSP33TAADNCL2XLHKEQ, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.900629124 (513h14m15.771712169s) [noise=true , address=145.79.227.45:4229 , piece_id=REFUVKI7DB3U464KOM3HP66SOHBEUD5C7QO4PHHFKWRIU3VOTFDA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.905117851 (513h14m15.776200896s) [noise=true , address=145.79.227.2:4115 , piece_id=3ANBT2RYXVTSEHSPOYR3LMUTCGYZR7WYYNVVJ2YQKHN6BEM5QUTA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.905595411 (513h14m15.776678456s) [noise=true , address=145.79.227.38:4978 , piece_id=ECI3U6UBNEKMNACECKBAQLK423S63YGUBIHGELZ75E7YN3UAMWVA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.906622075 (513h14m15.77770511s) [noise=true , address=145.79.227.36:4519 , piece_id=LUWN3PRDZQGW65O3K5SG4FIB5JBERIW5QWKYKYUTJUAQTMHABBOA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.923866802 (513h14m15.794949837s) [noise=true , address=145.79.227.47:4267 , piece_id=4RHTBMFLM3Q2DAJXG7CKQ3XGPOT2N7FH4I6NMWQ7K2C6HJG34ARA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.976109839 (513h14m15.847192884s) [noise=true , address=145.79.227.49:4338 , piece_id=HYQYAJAXR7QSD45JR35GBQEIP6LGKID2N2V5TX6CYFUDYKXNCB7A, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.068723638 (513h14m15.939806673s) [noise=true , address=77.90.31.19:4842 , piece_id=ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.256785211 (513h14m16.127868246s) [noise=true , address=45.91.116.13:4156 , piece_id=PEGHBWEQXJMYAHVVKD4F2EHFLUSJEIUTK4KXWB7ONGMBRU6VHDRA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.261236718 (513h14m16.132319753s) [noise=true , address=45.91.116.11:4119 , piece_id=QE3QOGTULK3F2JQCRMI6Y5ZDX5RZPKJGMVWUAGVO2GQN7FVQRP5Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.264852523 (513h14m16.135935568s) [noise=true , address=45.91.116.8:4046 , piece_id=IKWDIRO5HBIJ3XBNJQDI4VL7H3UG7BZPMTMXRUEMOEOUTDKOHWGA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.297360141 (513h14m16.168443196s) [noise=true , address=45.91.116.5:4298 , piece_id=565NHL6TPU5DDO2VHLHZPU4MF3O244WYEBWRAFEHJEXPU4ZGJ7XA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.373538565 (513h14m16.2446216s) [noise=true , address=45.91.116.9:4263 , piece_id=LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.373563493 (513h14m16.244646578s) [seg_pos={0 0}, upload=450357]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:43:16.373829158 (513h14m16.244912193s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:43:16.373837363 (513h14m16.244920398s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.373890735 (513h14m16.24497382s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection
Dec 27 00:43:16.373900754 (513h14m16.244983789s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.374121756 (513h14m16.245204791s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:55566->45.91.116.4:4937: use of closed network connection; context canceled
Dec 27 00:43:16.374128659 (513h14m16.245211704s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.374142144 (513h14m16.245225189s) [seg_pos={0 0}, upload=450357]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:43:16.374271821 (513h14m16.245354856s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:43:16.374277532 (513h14m16.245360597s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:43:16.671060053 (513h14m16.542143108s) [upload=450357]: Done uploading segments...
Dec 27 00:43:16.671085672 (513h14m16.542168717s) [upload=450357]: Upload finished. err: <nil>
Duration: 5.675122237s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
32.827849733s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:44:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 32.827877185s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [.X.........X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XXX] duration:
30.002658532s trace:
https://jaeger.dev.storj.io/trace/533cafd369c9ef95
FAIL (2025-12-27 00:43:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q) context deadline exceeded
======LOGS======
357]: Starting upload
Dec 27 00:43:13.647580027 (513h14m13.518663072s) [upload=450357]: Done starting upload
Dec 27 00:43:13.647712699 (513h14m13.518795734s) [upload=450357]: Uploading segments...
Dec 27 00:43:13.647829061 (513h14m13.518912096s) [upload=450357]: Got next segment. Inline: false
Dec 27 00:43:13.647886079 (513h14m13.518969124s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:43:13.647892362 (513h14m13.518975407s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:43:13.896495033 (513h14m13.767578078s) [seg_pos={0 0}, upload=450357]: Begin upload segment...
Dec 27 00:43:13.896558864 (513h14m13.767641899s) [seg_pos={0 0}, upload=450357]: Done begin upload segment.
Dec 27 00:43:13.896599492 (513h14m13.767682527s) [noise=true , address=145.79.227.36:4519 , piece_id=LUWN3PRDZQGW65O3K5SG4FIB5JBERIW5QWKYKYUTJUAQTMHABBOA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.896738566 (513h14m13.767821611s) [noise=true , address=45.91.116.13:4156 , piece_id=PEGHBWEQXJMYAHVVKD4F2EHFLUSJEIUTK4KXWB7ONGMBRU6VHDRA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897306407 (513h14m13.768389452s) [noise=true , address=45.91.116.5:4298 , piece_id=565NHL6TPU5DDO2VHLHZPU4MF3O244WYEBWRAFEHJEXPU4ZGJ7XA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897602832 (513h14m13.768685877s) [noise=true , address=145.79.227.50:4375 , piece_id=IEPVFXIPQCUNQLO2A2NR4TOVRYRVBNFYVOSP33TAADNCL2XLHKEQ, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.897926558 (513h14m13.769009593s) [noise=true , address=145.79.227.38:4978 , piece_id=ECI3U6UBNEKMNACECKBAQLK423S63YGUBIHGELZ75E7YN3UAMWVA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898244684 (513h14m13.769327719s) [noise=true , address=145.79.227.45:4229 , piece_id=REFUVKI7DB3U464KOM3HP66SOHBEUD5C7QO4PHHFKWRIU3VOTFDA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898517794 (513h14m13.769600839s) [noise=true , address=77.90.31.19:4842 , piece_id=ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.898801805 (513h14m13.76988484s) [noise=true , address=145.79.227.2:4115 , piece_id=3ANBT2RYXVTSEHSPOYR3LMUTCGYZR7WYYNVVJ2YQKHN6BEM5QUTA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.899116153 (513h14m13.770199188s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.900571314 (513h14m13.771654349s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.900724496 (513h14m13.771807531s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.902054269 (513h14m13.773137304s) [noise=true , address=45.91.116.9:4263 , piece_id=LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.903487227 (513h14m13.774570523s) [noise=true , address=145.79.227.47:4267 , piece_id=4RHTBMFLM3Q2DAJXG7CKQ3XGPOT2N7FH4I6NMWQ7K2C6HJG34ARA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.904408811 (513h14m13.775491847s) [noise=true , address=45.91.116.8:4046 , piece_id=IKWDIRO5HBIJ3XBNJQDI4VL7H3UG7BZPMTMXRUEMOEOUTDKOHWGA, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.907273937 (513h14m13.778356972s) [noise=true , address=145.79.227.49:4338 , piece_id=HYQYAJAXR7QSD45JR35GBQEIP6LGKID2N2V5TX6CYFUDYKXNCB7A, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:13.910120928 (513h14m13.781203963s) [noise=true , address=45.91.116.11:4119 , piece_id=QE3QOGTULK3F2JQCRMI6Y5ZDX5RZPKJGMVWUAGVO2GQN7FVQRP5Q, seg_pos={0 0}, upload=450357]: Uploading piece...
Dec 27 00:43:15.597774489 (513h14m15.468857524s) [upload=450357]: Next returned nil segment
Dec 27 00:43:15.597788597 (513h14m15.468871632s) [upload=450357]: Waiting for error group managing segments...
Dec 27 00:43:15.892620876 (513h14m15.763703911s) [noise=true , address=145.79.227.50:4375 , piece_id=IEPVFXIPQCUNQLO2A2NR4TOVRYRVBNFYVOSP33TAADNCL2XLHKEQ, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.900629124 (513h14m15.771712169s) [noise=true , address=145.79.227.45:4229 , piece_id=REFUVKI7DB3U464KOM3HP66SOHBEUD5C7QO4PHHFKWRIU3VOTFDA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.905117851 (513h14m15.776200896s) [noise=true , address=145.79.227.2:4115 , piece_id=3ANBT2RYXVTSEHSPOYR3LMUTCGYZR7WYYNVVJ2YQKHN6BEM5QUTA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.905595411 (513h14m15.776678456s) [noise=true , address=145.79.227.38:4978 , piece_id=ECI3U6UBNEKMNACECKBAQLK423S63YGUBIHGELZ75E7YN3UAMWVA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.906622075 (513h14m15.77770511s) [noise=true , address=145.79.227.36:4519 , piece_id=LUWN3PRDZQGW65O3K5SG4FIB5JBERIW5QWKYKYUTJUAQTMHABBOA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.923866802 (513h14m15.794949837s) [noise=true , address=145.79.227.47:4267 , piece_id=4RHTBMFLM3Q2DAJXG7CKQ3XGPOT2N7FH4I6NMWQ7K2C6HJG34ARA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:15.976109839 (513h14m15.847192884s) [noise=true , address=145.79.227.49:4338 , piece_id=HYQYAJAXR7QSD45JR35GBQEIP6LGKID2N2V5TX6CYFUDYKXNCB7A, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.068723638 (513h14m15.939806673s) [noise=true , address=77.90.31.19:4842 , piece_id=ITF2TKQJIRFR55E3NCBYCKO7A4766W7HX34WYN4IQOHLSFQOYMNA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.256785211 (513h14m16.127868246s) [noise=true , address=45.91.116.13:4156 , piece_id=PEGHBWEQXJMYAHVVKD4F2EHFLUSJEIUTK4KXWB7ONGMBRU6VHDRA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.261236718 (513h14m16.132319753s) [noise=true , address=45.91.116.11:4119 , piece_id=QE3QOGTULK3F2JQCRMI6Y5ZDX5RZPKJGMVWUAGVO2GQN7FVQRP5Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.264852523 (513h14m16.135935568s) [noise=true , address=45.91.116.8:4046 , piece_id=IKWDIRO5HBIJ3XBNJQDI4VL7H3UG7BZPMTMXRUEMOEOUTDKOHWGA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.297360141 (513h14m16.168443196s) [noise=true , address=45.91.116.5:4298 , piece_id=565NHL6TPU5DDO2VHLHZPU4MF3O244WYEBWRAFEHJEXPU4ZGJ7XA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.373538565 (513h14m16.2446216s) [noise=true , address=45.91.116.9:4263 , piece_id=LJKH72J236QWECWC3GZB22ORJATHDB7YXCGWW27GVYSPSO3JSK2Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: <nil>
Dec 27 00:43:16.373563493 (513h14m16.244646578s) [seg_pos={0 0}, upload=450357]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:43:16.373829158 (513h14m16.244912193s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:43:16.373837363 (513h14m16.244920398s) [noise=true , address=145.79.227.33:4869 , piece_id=5JYGM2EG3J5NG4RKDDXOHTRQPRWTYPAON7HYFIYSMHERCYCMYV5Q, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.373890735 (513h14m16.24497382s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43012->77.90.31.11:4553: use of closed network connection
Dec 27 00:43:16.373900754 (513h14m16.244983789s) [noise=true , address=77.90.31.11:4553 , piece_id=DQVLDVHGFFVG4Y6AZ73YEQRESB2ABESTOHEFR55ZAQZXE7724ICA, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.374121756 (513h14m16.245204791s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:55566->45.91.116.4:4937: use of closed network connection; context canceled
Dec 27 00:43:16.374128659 (513h14m16.245211704s) [noise=true , address=45.91.116.4:4937 , piece_id=2J6XO6RMO3RRXI3Z367XXWS4NBCR7V7KTOIRQT2QPKPV4PUUI4DQ, seg_pos={0 0}, upload=450357]: Long tail cancellation due to optimal threshold
Dec 27 00:43:16.374142144 (513h14m16.245225189s) [seg_pos={0 0}, upload=450357]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:43:16.374271821 (513h14m16.245354856s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:43:16.374277532 (513h14m16.245360597s) [upload=450357]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:43:16.671060053 (513h14m16.542143108s) [upload=450357]: Done uploading segments...
Dec 27 00:43:16.671085672 (513h14m16.542168717s) [upload=450357]: Upload finished. err: <nil>
Duration: 30.002682899s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
15.092291197s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:43:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 15.092331544s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [..X.........X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..XX] duration:
30.002997568s trace:
https://jaeger.dev.storj.io/trace/6d83912f5d8b6b6c
FAIL (2025-12-27 00:43:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: BUS6TG6HWIPJW4HYXDMDAN2XWNGJWMLU2DHX4XEONLUJAECZU3OA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ND6MHUCNNEYTWZ3XW3P5MJIN5HIZGK5JTUVOYIQ2XMN6F2P24PGQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7IRHHMSIDD6GWD4UMGNIATILQKTB2NDRHNC2IBCF7UMGZTWROBZA) context deadline exceeded
======LOGS======
ts...
Dec 27 00:42:42.091364071 (513h13m41.962447116s) [upload=450347]: Got next segment. Inline: false
Dec 27 00:42:42.091399859 (513h13m41.962482904s) [upload=450347]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:42:42.091404728 (513h13m41.962487773s) [upload=450347]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:42:42.342268495 (513h13m42.21335153s) [seg_pos={0 0}, upload=450347]: Begin upload segment...
Dec 27 00:42:42.342319924 (513h13m42.213402989s) [seg_pos={0 0}, upload=450347]: Done begin upload segment.
Dec 27 00:42:42.342357756 (513h13m42.213440791s) [noise=true , address=145.79.227.13:4802 , piece_id=WJE7URVZE664N6NTVHNY5EAS6WGBJEIBDKDH4O5VWSXBY4ILEQMA, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.342489637 (513h13m42.213572672s) [noise=true , address=45.91.116.12:4225 , piece_id=FK2W3BBKCOQHL6DPCCPVOEW4WFUARBYYAKGM7WTYXHRPLOXARK2Q, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.343048381 (513h13m42.214131416s) [noise=true , address=145.79.227.16:4909 , piece_id=4FBE4WQRY4IZFZSCS2ANNKFINX6R2MI6PBB7IR5GGSL2SXRLAHKA, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.343060343 (513h13m42.214143378s) [noise=true , address=145.79.227.3:4150 , piece_id=AQKW25LB72P576HZ4QJ3VHEUCH7UO2XUHPDK5SBGZ6FLPZAZ7ESQ, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.343322231 (513h13m42.214405277s) [noise=true , address=45.91.116.7:4010 , piece_id=BGO6WIDRRV6ZZVEHXA64KVCIDSREWOFUTBU7UU5WYUBGSLSASSNA, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.343488588 (513h13m42.214571623s) [noise=true , address=145.79.227.45:4231 , piece_id=3G47S6JB6I52OWIDALYUUFCFY22QAHCL6VHEJZDBGKDIQ3FPEFNQ, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.343615230 (513h13m42.214698265s) [noise=true , address=45.91.116.13:4156 , piece_id=ND6MHUCNNEYTWZ3XW3P5MJIN5HIZGK5JTUVOYIQ2XMN6F2P24PGQ, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.343884613 (513h13m42.214967658s) [noise=true , address=77.90.31.11:4554 , piece_id=7IRHHMSIDD6GWD4UMGNIATILQKTB2NDRHNC2IBCF7UMGZTWROBZA, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.343915743 (513h13m42.214998788s) [noise=true , address=145.79.227.41:4087 , piece_id=AB4RO5W3HZX4RQFKUVE6GL4MZ4ZB2BS5PG7RRF4NEDID2OJUTDFA, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.344202498 (513h13m42.215285533s) [noise=true , address=145.79.227.47:4266 , piece_id=GQ3G23AAJCKMPTDB4YZEQ37TKKSZJB2YIPEDZ4VFOCY7PXQUBDOA, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.344278272 (513h13m42.215361307s) [noise=true , address=45.91.116.11:4119 , piece_id=OTFNR42JKVOGTORTYM377LCPXBP4SEDGD3NJE357WMDUQT4EGLNQ, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.344459267 (513h13m42.215542302s) [noise=true , address=145.79.227.19:4077 , piece_id=YZ7Z6XBLJXUBWITUQO7DTM3REUAFTH6KMLUK3LOJT4HMYUG2JR3Q, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.344660500 (513h13m42.215743565s) [noise=true , address=145.79.227.37:4566 , piece_id=UBJOUMX7QWEWDWUK2ARAXOPC7DCWD7BJV6GSJYQGKJM4CKKJXFNQ, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.344714653 (513h13m42.215797698s) [noise=true , address=145.79.227.50:4373 , piece_id=RHY6L7BCNXVVNSX3RT6KFVOQ5Q4XRDYEDPNKRGVJEK4WG2S3QXKA, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.345047176 (513h13m42.216130212s) [noise=true , address=77.90.31.17:4770 , piece_id=BUS6TG6HWIPJW4HYXDMDAN2XWNGJWMLU2DHX4XEONLUJAECZU3OA, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:42.345487305 (513h13m42.21657034s) [noise=true , address=145.79.227.15:4875 , piece_id=XXHTWLBK6TIROQJTLRZ4DKISNW2K5TPJISHCFHUWLWO6FFMMRYDQ, seg_pos={0 0}, upload=450347]: Uploading piece...
Dec 27 00:42:44.183629293 (513h13m44.054712338s) [upload=450347]: Next returned nil segment
Dec 27 00:42:44.183645434 (513h13m44.054728479s) [upload=450347]: Waiting for error group managing segments...
Dec 27 00:42:44.496348080 (513h13m44.367431125s) [noise=true , address=145.79.227.45:4231 , piece_id=3G47S6JB6I52OWIDALYUUFCFY22QAHCL6VHEJZDBGKDIQ3FPEFNQ, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.504855459 (513h13m44.375938494s) [noise=true , address=145.79.227.37:4566 , piece_id=UBJOUMX7QWEWDWUK2ARAXOPC7DCWD7BJV6GSJYQGKJM4CKKJXFNQ, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.511074421 (513h13m44.382157466s) [noise=true , address=145.79.227.41:4087 , piece_id=AB4RO5W3HZX4RQFKUVE6GL4MZ4ZB2BS5PG7RRF4NEDID2OJUTDFA, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.511131970 (513h13m44.382215016s) [noise=true , address=145.79.227.15:4875 , piece_id=XXHTWLBK6TIROQJTLRZ4DKISNW2K5TPJISHCFHUWLWO6FFMMRYDQ, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.520211981 (513h13m44.391295016s) [noise=true , address=145.79.227.13:4802 , piece_id=WJE7URVZE664N6NTVHNY5EAS6WGBJEIBDKDH4O5VWSXBY4ILEQMA, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.524100825 (513h13m44.39518386s) [noise=true , address=77.90.31.17:4770 , piece_id=BUS6TG6HWIPJW4HYXDMDAN2XWNGJWMLU2DHX4XEONLUJAECZU3OA, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.524647386 (513h13m44.395730421s) [noise=true , address=145.79.227.19:4077 , piece_id=YZ7Z6XBLJXUBWITUQO7DTM3REUAFTH6KMLUK3LOJT4HMYUG2JR3Q, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.526101485 (513h13m44.39718452s) [noise=true , address=145.79.227.47:4266 , piece_id=GQ3G23AAJCKMPTDB4YZEQ37TKKSZJB2YIPEDZ4VFOCY7PXQUBDOA, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.584139386 (513h13m44.455222421s) [noise=true , address=77.90.31.11:4554 , piece_id=7IRHHMSIDD6GWD4UMGNIATILQKTB2NDRHNC2IBCF7UMGZTWROBZA, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.590257887 (513h13m44.461340922s) [noise=true , address=45.91.116.13:4156 , piece_id=ND6MHUCNNEYTWZ3XW3P5MJIN5HIZGK5JTUVOYIQ2XMN6F2P24PGQ, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.691148826 (513h13m44.562231861s) [noise=true , address=45.91.116.7:4010 , piece_id=BGO6WIDRRV6ZZVEHXA64KVCIDSREWOFUTBU7UU5WYUBGSLSASSNA, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.712390773 (513h13m44.583473808s) [noise=true , address=45.91.116.12:4225 , piece_id=FK2W3BBKCOQHL6DPCCPVOEW4WFUARBYYAKGM7WTYXHRPLOXARK2Q, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.845182225 (513h13m44.71626526s) [noise=true , address=145.79.227.50:4373 , piece_id=RHY6L7BCNXVVNSX3RT6KFVOQ5Q4XRDYEDPNKRGVJEK4WG2S3QXKA, seg_pos={0 0}, upload=450347]: Done uploading piece. err: <nil>
Dec 27 00:42:44.845207553 (513h13m44.716290598s) [seg_pos={0 0}, upload=450347]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:42:44.845518505 (513h13m44.71660155s) [noise=true , address=145.79.227.3:4150 , piece_id=AQKW25LB72P576HZ4QJ3VHEUCH7UO2XUHPDK5SBGZ6FLPZAZ7ESQ, seg_pos={0 0}, upload=450347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:42:44.845530017 (513h13m44.716613052s) [noise=true , address=145.79.227.3:4150 , piece_id=AQKW25LB72P576HZ4QJ3VHEUCH7UO2XUHPDK5SBGZ6FLPZAZ7ESQ, seg_pos={0 0}, upload=450347]: Long tail cancellation due to optimal threshold
Dec 27 00:42:44.845563390 (513h13m44.716646446s) [noise=true , address=45.91.116.11:4119 , piece_id=OTFNR42JKVOGTORTYM377LCPXBP4SEDGD3NJE357WMDUQT4EGLNQ, seg_pos={0 0}, upload=450347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:45358->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:45358->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45358->45.91.116.11:4119: use of closed network connection
Dec 27 00:42:44.845572237 (513h13m44.716655292s) [noise=true , address=45.91.116.11:4119 , piece_id=OTFNR42JKVOGTORTYM377LCPXBP4SEDGD3NJE357WMDUQT4EGLNQ, seg_pos={0 0}, upload=450347]: Long tail cancellation due to optimal threshold
Dec 27 00:42:44.845857231 (513h13m44.716940296s) [noise=true , address=145.79.227.16:4909 , piece_id=4FBE4WQRY4IZFZSCS2ANNKFINX6R2MI6PBB7IR5GGSL2SXRLAHKA, seg_pos={0 0}, upload=450347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:38444->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:38444->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38444->145.79.227.16:4909: use of closed network connection
Dec 27 00:42:44.845867069 (513h13m44.716950114s) [noise=true , address=145.79.227.16:4909 , piece_id=4FBE4WQRY4IZFZSCS2ANNKFINX6R2MI6PBB7IR5GGSL2SXRLAHKA, seg_pos={0 0}, upload=450347]: Long tail cancellation due to optimal threshold
Dec 27 00:42:44.845892137 (513h13m44.716975172s) [seg_pos={0 0}, upload=450347]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:42:44.845977078 (513h13m44.717060133s) [upload=450347]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:42:44.845985664 (513h13m44.71706871s) [upload=450347]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:42:45.133356937 (513h13m45.004439982s) [upload=450347]: Done uploading segments...
Dec 27 00:42:45.133375291 (513h13m45.004458326s) [upload=450347]: Upload finished. err: <nil>
Duration: 30.003028014s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
30.236404961s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:42:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 30.236432554s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [...X.........X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..X] duration:
30.002528006s trace:
https://jaeger.dev.storj.io/trace/3e7c4449981fb92
FAIL (2025-12-27 00:42:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ents...
Dec 27 00:41:56.729949923 (513h12m56.601032958s) [upload=450337]: Got next segment. Inline: false
Dec 27 00:41:56.731423759 (513h12m56.602506794s) [upload=450337]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:41:56.731435923 (513h12m56.602518968s) [upload=450337]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:41:56.976976364 (513h12m56.848059409s) [seg_pos={0 0}, upload=450337]: Begin upload segment...
Dec 27 00:41:56.977090010 (513h12m56.848173055s) [seg_pos={0 0}, upload=450337]: Done begin upload segment.
Dec 27 00:41:56.977145596 (513h12m56.848228631s) [noise=true , address=45.91.116.6:4974 , piece_id=JY5DU6GIK3GMVXC2ZTIOKJC4OHZJ64XPXEARBZOXO74WVWZKRZUA, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.977353913 (513h12m56.848436958s) [noise=true , address=145.79.227.46:4603 , piece_id=P7Q7L5DXC5HMFDGW66WBISLREWKTHBVFNPDWKWZLNKBLBPCAWWDA, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.978139358 (513h12m56.849222393s) [noise=true , address=145.79.227.49:4337 , piece_id=XTTQZ2KNK45O6K33LPW7KAUHN7N7HNFC6NPVOMRSG5XVRF2Y3Q5A, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.978242855 (513h12m56.84932589s) [noise=true , address=45.91.116.5:4297 , piece_id=S2F6U6QPRNO4FDFHS6HRIOE6EXQDP4FC3PJB6WH3HAVEBCWPJA6Q, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.978290466 (513h12m56.849373501s) [noise=true , address=45.91.116.11:4117 , piece_id=MWSLMYC4OVL7TK2EOOT52AKAAFMW6YEYI7YRYWXEZIO4SGZXHHNA, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.978768085 (513h12m56.84985115s) [noise=true , address=145.79.227.17:4972 , piece_id=NMWP6V44ODQJGOC5HUA47CUVADXF6WT4BPUGHDNLPL5FGRFZTFJA, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.978801489 (513h12m56.849884534s) [noise=true , address=145.79.227.33:4870 , piece_id=5FBWY66265JDXZCSBD4BSJUCEPAGYRAO2DKGWDJOFUO4SVEKU3MQ, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.979212051 (513h12m56.850295086s) [noise=true , address=45.91.116.7:4009 , piece_id=3LCSN7OC5Y4IRQDFVIQPYU5FMBWWNHO47SUOUJAIAC4T7Z4WEHMA, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.979283667 (513h12m56.850366702s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBZ4FVVD376CLSZOO6IWWKYDQHGFJ7IJ4PRBJOQS4I4OYGJB5OLQ, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.979649784 (513h12m56.850732819s) [noise=true , address=145.79.227.3:4150 , piece_id=ZWT4COYUT32HXZRSXTZMSBPVEW4FE4CY3VGQO5TOSKAGT6RFOVAA, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.979815660 (513h12m56.850898705s) [noise=true , address=77.90.31.11:4554 , piece_id=L6IXBKPXHN6ZAPQM7HGDAAODSKV7TOPCAXL6VLHA2VZJI52WZW5A, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.980295223 (513h12m56.851378258s) [noise=true , address=145.79.227.15:4874 , piece_id=KMSWSWXKW63PHAMGM422CGYRKPCQDDDNKUDEQDOYINRJ34S3Y4BQ, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.980920744 (513h12m56.852003779s) [noise=true , address=77.90.31.16:4736 , piece_id=JHJVYBTJZC5HJBBDYG5QMY3PFLFGXNVJHXZKJFADFK7NCWELOLBA, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.981442698 (513h12m56.852525733s) [noise=true , address=145.79.227.36:4517 , piece_id=M2TYEUFT5J4W7PA3U47XMEEXCZC7TE3KCA2STUO2KSF3KJ6IEFRQ, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.982201253 (513h12m56.853284298s) [noise=true , address=145.79.227.47:4266 , piece_id=KMWJ2WJ5ECL2N7NQAYEN37VN2VHCQDXP6AAPURNTJZURTFJWCSOA, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:56.982571567 (513h12m56.853654602s) [noise=true , address=145.79.227.41:4085 , piece_id=7RQ6OE22HF34M5G6WKZ6UFFX4FNYT7DUG5E42GSKI47OJZJ5JASA, seg_pos={0 0}, upload=450337]: Uploading piece...
Dec 27 00:41:58.696644404 (513h12m58.567727439s) [upload=450337]: Next returned nil segment
Dec 27 00:41:58.696665203 (513h12m58.567748238s) [upload=450337]: Waiting for error group managing segments...
Dec 27 00:41:58.979395008 (513h12m58.850478043s) [noise=true , address=145.79.227.17:4972 , piece_id=NMWP6V44ODQJGOC5HUA47CUVADXF6WT4BPUGHDNLPL5FGRFZTFJA, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:58.983456671 (513h12m58.854539706s) [noise=true , address=145.79.227.49:4337 , piece_id=XTTQZ2KNK45O6K33LPW7KAUHN7N7HNFC6NPVOMRSG5XVRF2Y3Q5A, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:58.987396062 (513h12m58.858479097s) [noise=true , address=145.79.227.46:4603 , piece_id=P7Q7L5DXC5HMFDGW66WBISLREWKTHBVFNPDWKWZLNKBLBPCAWWDA, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:58.995507508 (513h12m58.866590553s) [noise=true , address=145.79.227.36:4517 , piece_id=M2TYEUFT5J4W7PA3U47XMEEXCZC7TE3KCA2STUO2KSF3KJ6IEFRQ, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:58.998981232 (513h12m58.870064277s) [noise=true , address=145.79.227.47:4266 , piece_id=KMWJ2WJ5ECL2N7NQAYEN37VN2VHCQDXP6AAPURNTJZURTFJWCSOA, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:58.999613647 (513h12m58.870696682s) [noise=true , address=145.79.227.15:4874 , piece_id=KMSWSWXKW63PHAMGM422CGYRKPCQDDDNKUDEQDOYINRJ34S3Y4BQ, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:59.005703151 (513h12m58.876786196s) [noise=true , address=145.79.227.3:4150 , piece_id=ZWT4COYUT32HXZRSXTZMSBPVEW4FE4CY3VGQO5TOSKAGT6RFOVAA, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:59.006330987 (513h12m58.877414022s) [noise=true , address=145.79.227.41:4085 , piece_id=7RQ6OE22HF34M5G6WKZ6UFFX4FNYT7DUG5E42GSKI47OJZJ5JASA, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:59.187305556 (513h12m59.058388591s) [noise=true , address=77.90.31.11:4554 , piece_id=L6IXBKPXHN6ZAPQM7HGDAAODSKV7TOPCAXL6VLHA2VZJI52WZW5A, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:59.204631447 (513h12m59.075714482s) [noise=true , address=45.91.116.7:4009 , piece_id=3LCSN7OC5Y4IRQDFVIQPYU5FMBWWNHO47SUOUJAIAC4T7Z4WEHMA, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:59.258269220 (513h12m59.129352255s) [noise=true , address=45.91.116.6:4974 , piece_id=JY5DU6GIK3GMVXC2ZTIOKJC4OHZJ64XPXEARBZOXO74WVWZKRZUA, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:59.301028749 (513h12m59.172111785s) [noise=true , address=77.90.31.16:4736 , piece_id=JHJVYBTJZC5HJBBDYG5QMY3PFLFGXNVJHXZKJFADFK7NCWELOLBA, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:59.465447173 (513h12m59.336530208s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBZ4FVVD376CLSZOO6IWWKYDQHGFJ7IJ4PRBJOQS4I4OYGJB5OLQ, seg_pos={0 0}, upload=450337]: Done uploading piece. err: <nil>
Dec 27 00:41:59.465472001 (513h12m59.336555046s) [seg_pos={0 0}, upload=450337]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:41:59.465706648 (513h12m59.336789683s) [noise=true , address=45.91.116.5:4297 , piece_id=S2F6U6QPRNO4FDFHS6HRIOE6EXQDP4FC3PJB6WH3HAVEBCWPJA6Q, seg_pos={0 0}, upload=450337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:49234->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:49234->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49234->45.91.116.5:4297: use of closed network connection
Dec 27 00:41:59.465716186 (513h12m59.336799221s) [noise=true , address=45.91.116.5:4297 , piece_id=S2F6U6QPRNO4FDFHS6HRIOE6EXQDP4FC3PJB6WH3HAVEBCWPJA6Q, seg_pos={0 0}, upload=450337]: Long tail cancellation due to optimal threshold
Dec 27 00:41:59.465786660 (513h12m59.336869695s) [noise=true , address=145.79.227.33:4870 , piece_id=5FBWY66265JDXZCSBD4BSJUCEPAGYRAO2DKGWDJOFUO4SVEKU3MQ, seg_pos={0 0}, upload=450337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 27 00:41:59.465823240 (513h12m59.336906285s) [noise=true , address=145.79.227.33:4870 , piece_id=5FBWY66265JDXZCSBD4BSJUCEPAGYRAO2DKGWDJOFUO4SVEKU3MQ, seg_pos={0 0}, upload=450337]: Long tail cancellation due to optimal threshold
Dec 27 00:41:59.465923811 (513h12m59.337006856s) [noise=true , address=45.91.116.11:4117 , piece_id=MWSLMYC4OVL7TK2EOOT52AKAAFMW6YEYI7YRYWXEZIO4SGZXHHNA, seg_pos={0 0}, upload=450337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:33686->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:33686->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33686->45.91.116.11:4117: use of closed network connection
Dec 27 00:41:59.465931886 (513h12m59.337014921s) [noise=true , address=45.91.116.11:4117 , piece_id=MWSLMYC4OVL7TK2EOOT52AKAAFMW6YEYI7YRYWXEZIO4SGZXHHNA, seg_pos={0 0}, upload=450337]: Long tail cancellation due to optimal threshold
Dec 27 00:41:59.465955001 (513h12m59.337038036s) [seg_pos={0 0}, upload=450337]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:41:59.466043829 (513h12m59.337126864s) [upload=450337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:41:59.466048519 (513h12m59.337131554s) [upload=450337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:41:59.757219085 (513h12m59.62830213s) [upload=450337]: Done uploading segments...
Dec 27 00:41:59.757243191 (513h12m59.628326236s) [upload=450337]: Upload finished. err: <nil>
Duration: 30.002554407s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
37.430024722s trace:
https://jaeger.dev.storj.io/trace/2c300281cb8f70aa
FAIL (2025-12-27 00:41:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
{0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626091168 (513h2m6.497174213s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626240413 (513h2m6.497323448s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.626666644 (513h2m6.497749689s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627048852 (513h2m6.498131887s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627446769 (513h2m6.498529804s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.627800362 (513h2m6.498883397s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.630782842 (513h2m6.501865887s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631264769 (513h2m6.502347804s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.631924075 (513h2m6.50300712s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633033637 (513h2m6.504116682s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.633969429 (513h2m6.505052464s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.634721000 (513h2m6.505804045s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636381061 (513h2m6.507464096s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.636959162 (513h2m6.508042207s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.638503492 (513h2m6.509586537s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:06.639678710 (513h2m6.510761745s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Uploading piece...
Dec 27 00:31:08.021948240 (513h2m7.893031275s) [upload=450180]: Next returned nil segment
Dec 27 00:31:08.021960633 (513h2m7.893043668s) [upload=450180]: Waiting for error group managing segments...
Dec 27 00:31:08.401952345 (513h2m8.27303538s) [noise=true , address=64.49.13.140:4507 , piece_id=ZYD67ZW3GHMWCVCZCYEHSAGAV7LPOJCWVJ65WGRYYK6OKHAB53EQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.521100806 (513h2m8.392183861s) [noise=true , address=145.79.227.2:4117 , piece_id=ZNAKRUW7E63AA7HHTJTO6JJLZ7IYUUO4F4FEIY6AOED4H2GF5QOQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.530568132 (513h2m8.401651167s) [noise=true , address=145.79.227.47:4283 , piece_id=23G75TZR5XEQVCCO4CELPNMXEEMPOIXNNWA2T37PFD54UDISFZZA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.546921841 (513h2m8.418004876s) [noise=true , address=145.79.227.34:4911 , piece_id=QWAJ2G5RB2UNWAHK6L6NTTWIKZOXHTESMM3BNFWE4F6NJSTT5F7Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.548756986 (513h2m8.419840021s) [noise=true , address=5.231.73.2:4026 , piece_id=WEQIF5NXSHOUPZ3Z2OS66P62G442JW6DZWQDHZKKVCD2KG3FBHKA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.553151353 (513h2m8.424234388s) [noise=true , address=145.79.227.41:4115 , piece_id=OUCVO4JH335OPJUGQYCP7NF5BO4RGOBSYW7P3ZOJKWP7I4GM42PA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.702119232 (513h2m8.573202267s) [noise=true , address=77.90.31.19:4847 , piece_id=LRZ6PAVJGDMRGB4X4WA6VIO2CDGKN6EMAR3PX4WGSKQ3KDOJARMQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.727290801 (513h2m8.598373836s) [noise=true , address=145.79.227.34:4910 , piece_id=3BPIHKPPR5INTHE76QOHGDESLV23QYUO7ZVIGJ4ZPOWION4ENCUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.754961800 (513h2m8.626044845s) [noise=true , address=77.90.31.13:4639 , piece_id=3FISGLEITMYHNRQ6YVH3SMU4II2GSSWXFMAKDNN2D5N4JZDDXXUA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.782979096 (513h2m8.654062142s) [noise=true , address=77.90.31.10:4521 , piece_id=NHU3KHNF3E5U7HPFBIFFIJGRIYEFRQ7PJ35DMNEWKPG5C46WH52Q, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:08.961516900 (513h2m8.832599946s) [noise=true , address=77.90.31.11:4560 , piece_id=K4QGSMMFEYT4QDBDKGKKQH7KB2IW72RLHD5E3EDFDXIYNBEY3WWA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.405709897 (513h2m9.276792943s) [noise=true , address=145.79.227.30:4772 , piece_id=PC453PVQGTHEIBA4ZPKEWKVT4DFV6QOGYOOSQ22IQ2FM34YBCZIA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.409575458 (513h2m9.280658493s) [noise=true , address=145.79.227.44:4219 , piece_id=LP7YUIHYQY5PF3ZHOXSUAIS7GLLWGTKWQEFFKZCDYTMSQF3WS2VQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449865184 (513h2m9.320948229s) [noise=true , address=145.79.227.44:4200 , piece_id=MMTHRS7SELSSETQGBK74DPSGJWR445OHVAXKQN4NBVA44WPPEUWQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: <nil>
Dec 27 00:31:09.449892176 (513h2m9.320975211s) [seg_pos={0 0}, upload=450180]: Segment reached optimal threshold of 14 pieces.
Dec 27 00:31:09.450243304 (513h2m9.32132636s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450253524 (513h2m9.321336559s) [noise=true , address=145.79.227.40:4073 , piece_id=Q4OY2EAKCANPPKXI32ZIPQTKUW6HRAA6XDW37MFXNVPZL66XM6IA, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450328296 (513h2m9.321411351s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pFjszrBBZvWsoeXM43kLgX7uNqk8iwSG8nsE1zbj77GVfQcKn): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450339647 (513h2m9.321422692s) [noise=true , address=45.91.116.13:4179 , piece_id=6GPWJM6QPXWOF2KIIPS3ZMLG4LCZLMWWHEKA76OQXDFU7FG345CQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450629760 (513h2m9.321712795s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->64.49.13.169:4708: use of closed network connection
Dec 27 00:31:09.450659597 (513h2m9.321742642s) [noise=true , address=64.49.13.169:4708 , piece_id=347BAFAJ42MFL42M2RZ4SIILLJRYXSKT5GDOOT6YGVABXQ57WQZQ, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450796949 (513h2m9.321879984s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF
Dec 27 00:31:09.450805535 (513h2m9.32188859s) [noise=true , address=145.79.227.43:4184 , piece_id=LVBIZIXOZHULUXCWS6TSMNKQO2MKG7WTZFHON374I2FZKMBALC4A, seg_pos={0 0}, upload=450180]: Long tail cancellation due to optimal threshold
Dec 27 00:31:09.450848216 (513h2m9.321931261s) [seg_pos={0 0}, upload=450180]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 00:31:09.450947074 (513h2m9.322030119s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:31:09.450954138 (513h2m9.322037183s) [upload=450180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:31:09.743160037 (513h2m9.614243072s) [upload=450180]: Done uploading segments...
Dec 27 00:31:09.743178352 (513h2m9.614261387s) [upload=450180]: Upload finished. err: <nil>
Duration: 37.43005476s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [....X.........X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X..] duration:
6.20647659s trace:
https://jaeger.dev.storj.io/trace/11bd0672458f6d04
FAIL (2025-12-27 00:41:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 3NZYAGEGTGCQAEVCNXTK6LSMOYK2IQSMCNRUPHMMBJFLKHQLXHIQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 7URUCXR3WWS4ELV2AQAANMBFTAWNZHSCCFAEIMDOKXDS7J5KRVUQ) context deadline exceeded
======LOGS======
Dec 27 00:39:45.786480873 (513h10m45.657563918s) [upload=450303]: Starting upload
Dec 27 00:39:45.786582045 (513h10m45.65766511s) [upload=450303]: Done starting upload
Dec 27 00:39:45.787016904 (513h10m45.658099949s) [upload=450303]: Uploading segments...
Dec 27 00:39:45.787414741 (513h10m45.658497786s) [upload=450303]: Got next segment. Inline: false
Dec 27 00:39:45.787460719 (513h10m45.658543764s) [upload=450303]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:39:45.787466270 (513h10m45.658549315s) [upload=450303]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:39:46.034614484 (513h10m45.905697519s) [seg_pos={0 0}, upload=450303]: Begin upload segment...
Dec 27 00:39:46.034655301 (513h10m45.905738346s) [seg_pos={0 0}, upload=450303]: Done begin upload segment.
Dec 27 00:39:46.034685529 (513h10m45.905768564s) [noise=true , address=45.91.116.5:4300 , piece_id=WMXMUYEF34ONHQMTKAYXTOJKCYB7GHM5FR42DUTFIMGW5QTEVKFQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.037881384 (513h10m45.908964419s) [noise=true , address=45.91.116.10:4082 , piece_id=QLBPCZZACW244PFW7KKRIXS22RBMO4OISLDGXWVFVCWURX3ZWOFA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.039330834 (513h10m45.910413869s) [noise=true , address=145.79.227.11:4730 , piece_id=RXN3LBMXK3N4Z6PHWIVQDDNLAJIBS5CMMQ3UYMQ5KR7SD574SNFQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.040463050 (513h10m45.911546095s) [noise=true , address=45.91.116.11:4117 , piece_id=LLTL6PL34GMMCIJBT23JQ5PPGDLL5W5HSFUPWE4VKLH6RWJKJ4AA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.041492861 (513h10m45.912575906s) [noise=true , address=45.91.116.9:4264 , piece_id=7URUCXR3WWS4ELV2AQAANMBFTAWNZHSCCFAEIMDOKXDS7J5KRVUQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.042656977 (513h10m45.913740012s) [noise=true , address=145.79.227.48:4301 , piece_id=FSKUGEXFQLGQ2S2YXHBJK73LSVOZXWUWGNGCUQ7COBJU2KSPHMUA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.042690661 (513h10m45.913773706s) [noise=true , address=145.79.227.37:4565 , piece_id=TVD5S3GYKRMFJ64GJPBLCZFATKSMXKEB4UF5G7VKJSR3LCF4H4XA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.043695805 (513h10m45.91477884s) [noise=true , address=145.79.227.51:4411 , piece_id=3PVQGDIRK5BJF4AD3ML3Z4FV3IWQZ3TRUUVIWE4LE5D2LFKJSWSQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.043868103 (513h10m45.914951138s) [noise=true , address=77.90.31.10:4517 , piece_id=XKHU6H6W2W5UEVHKQAPC4JOQXQ4EIBIMV4B3Z6XTF6ZLIZDNKDHQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.044801541 (513h10m45.915884576s) [noise=true , address=145.79.227.50:4375 , piece_id=SODNED6KCTGEOEM4FFOYAKIFAZWZXQST2RB4PL3QA7FJPCYKGKZQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.045263370 (513h10m45.916346405s) [noise=true , address=145.79.227.45:4231 , piece_id=A72TX2BJVYDN34NWUVDZ7KK3QZC5ZTSCNGB5XPVKGN6C4HUWSXVA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.046375698 (513h10m45.917458743s) [noise=true , address=145.79.227.13:4804 , piece_id=SXXWKFARZGBTAZRNVQWLFPMESJRSPJKORKXTQBPIRNYBHLXWEOMA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.046827088 (513h10m45.917910123s) [noise=true , address=45.91.116.12:4225 , piece_id=3NZYAGEGTGCQAEVCNXTK6LSMOYK2IQSMCNRUPHMMBJFLKHQLXHIQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.047514907 (513h10m45.918597972s) [noise=true , address=145.79.227.38:4978 , piece_id=OGLRLH2MCFXRARNPT3NWGRD7PO42CZHT67E657JHMM5ZG623U5OA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.048112816 (513h10m45.919195851s) [noise=true , address=45.91.116.4:4937 , piece_id=RE3LVBJIDSXSTY2P2QXLKHQNVLXEHWUITOJRCWEY3XQY2N6FHWXQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.048669235 (513h10m45.91975227s) [noise=true , address=45.91.116.7:4011 , piece_id=FCH3NGGYYXIZVOYHIW5JJV34PELCEBLPHOBPPET67224K6VABLAA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:47.754404463 (513h10m47.625487498s) [upload=450303]: Next returned nil segment
Dec 27 00:39:47.754443157 (513h10m47.625526202s) [upload=450303]: Waiting for error group managing segments...
Dec 27 00:39:48.036912636 (513h10m47.907995671s) [noise=true , address=145.79.227.11:4730 , piece_id=RXN3LBMXK3N4Z6PHWIVQDDNLAJIBS5CMMQ3UYMQ5KR7SD574SNFQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.037524491 (513h10m47.908607536s) [noise=true , address=145.79.227.48:4301 , piece_id=FSKUGEXFQLGQ2S2YXHBJK73LSVOZXWUWGNGCUQ7COBJU2KSPHMUA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.048855426 (513h10m47.919938471s) [noise=true , address=145.79.227.50:4375 , piece_id=SODNED6KCTGEOEM4FFOYAKIFAZWZXQST2RB4PL3QA7FJPCYKGKZQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.052168044 (513h10m47.923251089s) [noise=true , address=145.79.227.45:4231 , piece_id=A72TX2BJVYDN34NWUVDZ7KK3QZC5ZTSCNGB5XPVKGN6C4HUWSXVA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.053354723 (513h10m47.924437768s) [noise=true , address=145.79.227.37:4565 , piece_id=TVD5S3GYKRMFJ64GJPBLCZFATKSMXKEB4UF5G7VKJSR3LCF4H4XA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.059428338 (513h10m47.930511373s) [noise=true , address=145.79.227.13:4804 , piece_id=SXXWKFARZGBTAZRNVQWLFPMESJRSPJKORKXTQBPIRNYBHLXWEOMA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.062070509 (513h10m47.933153544s) [noise=true , address=145.79.227.51:4411 , piece_id=3PVQGDIRK5BJF4AD3ML3Z4FV3IWQZ3TRUUVIWE4LE5D2LFKJSWSQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.062340644 (513h10m47.933423689s) [noise=true , address=145.79.227.38:4978 , piece_id=OGLRLH2MCFXRARNPT3NWGRD7PO42CZHT67E657JHMM5ZG623U5OA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.238832081 (513h10m48.109915126s) [noise=true , address=77.90.31.10:4517 , piece_id=XKHU6H6W2W5UEVHKQAPC4JOQXQ4EIBIMV4B3Z6XTF6ZLIZDNKDHQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.259527186 (513h10m48.130610221s) [noise=true , address=45.91.116.10:4082 , piece_id=QLBPCZZACW244PFW7KKRIXS22RBMO4OISLDGXWVFVCWURX3ZWOFA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.302073639 (513h10m48.173156694s) [noise=true , address=45.91.116.9:4264 , piece_id=7URUCXR3WWS4ELV2AQAANMBFTAWNZHSCCFAEIMDOKXDS7J5KRVUQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.347234991 (513h10m48.218318026s) [noise=true , address=45.91.116.12:4225 , piece_id=3NZYAGEGTGCQAEVCNXTK6LSMOYK2IQSMCNRUPHMMBJFLKHQLXHIQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.352875811 (513h10m48.223958846s) [noise=true , address=45.91.116.4:4937 , piece_id=RE3LVBJIDSXSTY2P2QXLKHQNVLXEHWUITOJRCWEY3XQY2N6FHWXQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.352895378 (513h10m48.223978413s) [seg_pos={0 0}, upload=450303]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:39:48.353111229 (513h10m48.224194264s) [noise=true , address=45.91.116.5:4300 , piece_id=WMXMUYEF34ONHQMTKAYXTOJKCYB7GHM5FR42DUTFIMGW5QTEVKFQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:39:48.353121749 (513h10m48.224204784s) [noise=true , address=45.91.116.5:4300 , piece_id=WMXMUYEF34ONHQMTKAYXTOJKCYB7GHM5FR42DUTFIMGW5QTEVKFQ, seg_pos={0 0}, upload=450303]: Long tail cancellation due to optimal threshold
Dec 27 00:39:48.353163880 (513h10m48.224246925s) [noise=true , address=45.91.116.11:4117 , piece_id=LLTL6PL34GMMCIJBT23JQ5PPGDLL5W5HSFUPWE4VKLH6RWJKJ4AA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 27 00:39:48.353202173 (513h10m48.224285208s) [noise=true , address=45.91.116.11:4117 , piece_id=LLTL6PL34GMMCIJBT23JQ5PPGDLL5W5HSFUPWE4VKLH6RWJKJ4AA, seg_pos={0 0}, upload=450303]: Long tail cancellation due to optimal threshold
Dec 27 00:39:48.353262517 (513h10m48.224345552s) [noise=true , address=45.91.116.7:4011 , piece_id=FCH3NGGYYXIZVOYHIW5JJV34PELCEBLPHOBPPET67224K6VABLAA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 27 00:39:48.353269621 (513h10m48.224352656s) [noise=true , address=45.91.116.7:4011 , piece_id=FCH3NGGYYXIZVOYHIW5JJV34PELCEBLPHOBPPET67224K6VABLAA, seg_pos={0 0}, upload=450303]: Long tail cancellation due to optimal threshold
Dec 27 00:39:48.353292725 (513h10m48.22437576s) [seg_pos={0 0}, upload=450303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:39:48.353366244 (513h10m48.22444928s) [upload=450303]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:39:48.353399137 (513h10m48.224482172s) [upload=450303]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:39:48.645950194 (513h10m48.517033259s) [upload=450303]: Done uploading segments...
Dec 27 00:39:48.645973808 (513h10m48.517056843s) [upload=450303]: Upload finished. err: <nil>
Duration: 6.206503072s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.....X.........X.......X.XX.XX.XXXX..X.XXX.X.X..X..X...XX.X.] duration:
6.159212089s trace:
https://jaeger.dev.storj.io/trace/11bd0672458f6d04
FAIL (2025-12-27 00:40:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 3NZYAGEGTGCQAEVCNXTK6LSMOYK2IQSMCNRUPHMMBJFLKHQLXHIQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 7URUCXR3WWS4ELV2AQAANMBFTAWNZHSCCFAEIMDOKXDS7J5KRVUQ) context deadline exceeded
======LOGS======
Dec 27 00:39:45.786480873 (513h10m45.657563918s) [upload=450303]: Starting upload
Dec 27 00:39:45.786582045 (513h10m45.65766511s) [upload=450303]: Done starting upload
Dec 27 00:39:45.787016904 (513h10m45.658099949s) [upload=450303]: Uploading segments...
Dec 27 00:39:45.787414741 (513h10m45.658497786s) [upload=450303]: Got next segment. Inline: false
Dec 27 00:39:45.787460719 (513h10m45.658543764s) [upload=450303]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 00:39:45.787466270 (513h10m45.658549315s) [upload=450303]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 00:39:46.034614484 (513h10m45.905697519s) [seg_pos={0 0}, upload=450303]: Begin upload segment...
Dec 27 00:39:46.034655301 (513h10m45.905738346s) [seg_pos={0 0}, upload=450303]: Done begin upload segment.
Dec 27 00:39:46.034685529 (513h10m45.905768564s) [noise=true , address=45.91.116.5:4300 , piece_id=WMXMUYEF34ONHQMTKAYXTOJKCYB7GHM5FR42DUTFIMGW5QTEVKFQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.037881384 (513h10m45.908964419s) [noise=true , address=45.91.116.10:4082 , piece_id=QLBPCZZACW244PFW7KKRIXS22RBMO4OISLDGXWVFVCWURX3ZWOFA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.039330834 (513h10m45.910413869s) [noise=true , address=145.79.227.11:4730 , piece_id=RXN3LBMXK3N4Z6PHWIVQDDNLAJIBS5CMMQ3UYMQ5KR7SD574SNFQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.040463050 (513h10m45.911546095s) [noise=true , address=45.91.116.11:4117 , piece_id=LLTL6PL34GMMCIJBT23JQ5PPGDLL5W5HSFUPWE4VKLH6RWJKJ4AA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.041492861 (513h10m45.912575906s) [noise=true , address=45.91.116.9:4264 , piece_id=7URUCXR3WWS4ELV2AQAANMBFTAWNZHSCCFAEIMDOKXDS7J5KRVUQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.042656977 (513h10m45.913740012s) [noise=true , address=145.79.227.48:4301 , piece_id=FSKUGEXFQLGQ2S2YXHBJK73LSVOZXWUWGNGCUQ7COBJU2KSPHMUA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.042690661 (513h10m45.913773706s) [noise=true , address=145.79.227.37:4565 , piece_id=TVD5S3GYKRMFJ64GJPBLCZFATKSMXKEB4UF5G7VKJSR3LCF4H4XA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.043695805 (513h10m45.91477884s) [noise=true , address=145.79.227.51:4411 , piece_id=3PVQGDIRK5BJF4AD3ML3Z4FV3IWQZ3TRUUVIWE4LE5D2LFKJSWSQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.043868103 (513h10m45.914951138s) [noise=true , address=77.90.31.10:4517 , piece_id=XKHU6H6W2W5UEVHKQAPC4JOQXQ4EIBIMV4B3Z6XTF6ZLIZDNKDHQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.044801541 (513h10m45.915884576s) [noise=true , address=145.79.227.50:4375 , piece_id=SODNED6KCTGEOEM4FFOYAKIFAZWZXQST2RB4PL3QA7FJPCYKGKZQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.045263370 (513h10m45.916346405s) [noise=true , address=145.79.227.45:4231 , piece_id=A72TX2BJVYDN34NWUVDZ7KK3QZC5ZTSCNGB5XPVKGN6C4HUWSXVA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.046375698 (513h10m45.917458743s) [noise=true , address=145.79.227.13:4804 , piece_id=SXXWKFARZGBTAZRNVQWLFPMESJRSPJKORKXTQBPIRNYBHLXWEOMA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.046827088 (513h10m45.917910123s) [noise=true , address=45.91.116.12:4225 , piece_id=3NZYAGEGTGCQAEVCNXTK6LSMOYK2IQSMCNRUPHMMBJFLKHQLXHIQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.047514907 (513h10m45.918597972s) [noise=true , address=145.79.227.38:4978 , piece_id=OGLRLH2MCFXRARNPT3NWGRD7PO42CZHT67E657JHMM5ZG623U5OA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.048112816 (513h10m45.919195851s) [noise=true , address=45.91.116.4:4937 , piece_id=RE3LVBJIDSXSTY2P2QXLKHQNVLXEHWUITOJRCWEY3XQY2N6FHWXQ, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:46.048669235 (513h10m45.91975227s) [noise=true , address=45.91.116.7:4011 , piece_id=FCH3NGGYYXIZVOYHIW5JJV34PELCEBLPHOBPPET67224K6VABLAA, seg_pos={0 0}, upload=450303]: Uploading piece...
Dec 27 00:39:47.754404463 (513h10m47.625487498s) [upload=450303]: Next returned nil segment
Dec 27 00:39:47.754443157 (513h10m47.625526202s) [upload=450303]: Waiting for error group managing segments...
Dec 27 00:39:48.036912636 (513h10m47.907995671s) [noise=true , address=145.79.227.11:4730 , piece_id=RXN3LBMXK3N4Z6PHWIVQDDNLAJIBS5CMMQ3UYMQ5KR7SD574SNFQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.037524491 (513h10m47.908607536s) [noise=true , address=145.79.227.48:4301 , piece_id=FSKUGEXFQLGQ2S2YXHBJK73LSVOZXWUWGNGCUQ7COBJU2KSPHMUA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.048855426 (513h10m47.919938471s) [noise=true , address=145.79.227.50:4375 , piece_id=SODNED6KCTGEOEM4FFOYAKIFAZWZXQST2RB4PL3QA7FJPCYKGKZQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.052168044 (513h10m47.923251089s) [noise=true , address=145.79.227.45:4231 , piece_id=A72TX2BJVYDN34NWUVDZ7KK3QZC5ZTSCNGB5XPVKGN6C4HUWSXVA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.053354723 (513h10m47.924437768s) [noise=true , address=145.79.227.37:4565 , piece_id=TVD5S3GYKRMFJ64GJPBLCZFATKSMXKEB4UF5G7VKJSR3LCF4H4XA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.059428338 (513h10m47.930511373s) [noise=true , address=145.79.227.13:4804 , piece_id=SXXWKFARZGBTAZRNVQWLFPMESJRSPJKORKXTQBPIRNYBHLXWEOMA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.062070509 (513h10m47.933153544s) [noise=true , address=145.79.227.51:4411 , piece_id=3PVQGDIRK5BJF4AD3ML3Z4FV3IWQZ3TRUUVIWE4LE5D2LFKJSWSQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.062340644 (513h10m47.933423689s) [noise=true , address=145.79.227.38:4978 , piece_id=OGLRLH2MCFXRARNPT3NWGRD7PO42CZHT67E657JHMM5ZG623U5OA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.238832081 (513h10m48.109915126s) [noise=true , address=77.90.31.10:4517 , piece_id=XKHU6H6W2W5UEVHKQAPC4JOQXQ4EIBIMV4B3Z6XTF6ZLIZDNKDHQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.259527186 (513h10m48.130610221s) [noise=true , address=45.91.116.10:4082 , piece_id=QLBPCZZACW244PFW7KKRIXS22RBMO4OISLDGXWVFVCWURX3ZWOFA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.302073639 (513h10m48.173156694s) [noise=true , address=45.91.116.9:4264 , piece_id=7URUCXR3WWS4ELV2AQAANMBFTAWNZHSCCFAEIMDOKXDS7J5KRVUQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.347234991 (513h10m48.218318026s) [noise=true , address=45.91.116.12:4225 , piece_id=3NZYAGEGTGCQAEVCNXTK6LSMOYK2IQSMCNRUPHMMBJFLKHQLXHIQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.352875811 (513h10m48.223958846s) [noise=true , address=45.91.116.4:4937 , piece_id=RE3LVBJIDSXSTY2P2QXLKHQNVLXEHWUITOJRCWEY3XQY2N6FHWXQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: <nil>
Dec 27 00:39:48.352895378 (513h10m48.223978413s) [seg_pos={0 0}, upload=450303]: Segment reached optimal threshold of 13 pieces.
Dec 27 00:39:48.353111229 (513h10m48.224194264s) [noise=true , address=45.91.116.5:4300 , piece_id=WMXMUYEF34ONHQMTKAYXTOJKCYB7GHM5FR42DUTFIMGW5QTEVKFQ, seg_pos={0 0}, upload=450303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 27 00:39:48.353121749 (513h10m48.224204784s) [noise=true , address=45.91.116.5:4300 , piece_id=WMXMUYEF34ONHQMTKAYXTOJKCYB7GHM5FR42DUTFIMGW5QTEVKFQ, seg_pos={0 0}, upload=450303]: Long tail cancellation due to optimal threshold
Dec 27 00:39:48.353163880 (513h10m48.224246925s) [noise=true , address=45.91.116.11:4117 , piece_id=LLTL6PL34GMMCIJBT23JQ5PPGDLL5W5HSFUPWE4VKLH6RWJKJ4AA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 27 00:39:48.353202173 (513h10m48.224285208s) [noise=true , address=45.91.116.11:4117 , piece_id=LLTL6PL34GMMCIJBT23JQ5PPGDLL5W5HSFUPWE4VKLH6RWJKJ4AA, seg_pos={0 0}, upload=450303]: Long tail cancellation due to optimal threshold
Dec 27 00:39:48.353262517 (513h10m48.224345552s) [noise=true , address=45.91.116.7:4011 , piece_id=FCH3NGGYYXIZVOYHIW5JJV34PELCEBLPHOBPPET67224K6VABLAA, seg_pos={0 0}, upload=450303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 27 00:39:48.353269621 (513h10m48.224352656s) [noise=true , address=45.91.116.7:4011 , piece_id=FCH3NGGYYXIZVOYHIW5JJV34PELCEBLPHOBPPET67224K6VABLAA, seg_pos={0 0}, upload=450303]: Long tail cancellation due to optimal threshold
Dec 27 00:39:48.353292725 (513h10m48.22437576s) [seg_pos={0 0}, upload=450303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 00:39:48.353366244 (513h10m48.22444928s) [upload=450303]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 00:39:48.353399137 (513h10m48.224482172s) [upload=450303]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 00:39:48.645950194 (513h10m48.517033259s) [upload=450303]: Done uploading segments...
Dec 27 00:39:48.645973808 (513h10m48.517056843s) [upload=450303]: Upload finished. err: <nil>
Duration: 6.159243428s
core
uplink
us-select-1
|