Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.....................................X......X...X...] duration: 15.4260807s trace: https://jaeger.dev.storj.io/trace/70b0cb39dd42d67b FAIL (2025-12-31 17:03:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.156432368 (625h31m18.027515413s) [noise=true , address=77.90.31.15:4710 , piece_id=AOXLJRW7BEWM7I3VFR64ONRKI3SX5SSUDH47SAXA3IAUPSRPZAOQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157072997 (625h31m18.028156042s) [noise=true , address=145.79.227.52:4459 , piece_id=OIZHPVYAWOUFB6EGGW65AYJA7CD2OJHPP5BV7X2BKON6OFSULSRA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157213074 (625h31m18.028296159s) [noise=true , address=77.90.31.15:4723 , piece_id=CKPWFCY5N3ZCZUBILOS3RZOS6AEFLWN4GH65L25TVYZAXZGQMWVA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157552671 (625h31m18.028635716s) [noise=true , address=77.90.31.17:4786 , piece_id=C6HSG4XIOQDNTB2RJTJOMFYOMDFDRF4MG2XFAPRSPS726NNBUTLQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157787158 (625h31m18.028870203s) [noise=true , address=145.79.227.40:4072 , piece_id=UE56KE2MGKKROEX4M75BSK4WXD2IH2TDL3DVVFVDPTPX2HRQI4OQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158098741 (625h31m18.029181786s) [noise=true , address=77.90.31.13:4625 , piece_id=DPFGEZR5J2WRRNTISDYXJKWUTZSTRYDY2BS25HTOXQHVRZL52TRA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158386689 (625h31m18.029469734s) [noise=true , address=145.79.227.49:4343 , piece_id=UFWFBVKIWXCN6QMKXV3D3DCLPEV3DXW27N3J5AK3JWSJ2LXQZAIA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158597901 (625h31m18.029680946s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.159008944 (625h31m18.030091989s) [noise=true , address=145.79.227.30:4779 , piece_id=HIJQGU3DHFJHVR3ISVSWPZBDX5Y2Q5N4LTUM43GOXCOURUV2AJTQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.159761006 (625h31m18.030844051s) [noise=true , address=145.79.227.49:4345 , piece_id=RIPCK32N6WLUQYS7EPK6TSEEAQH2DUVO3JZ4VVW3SJSCRCRWZE2A, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.160164584 (625h31m18.03124763s) [noise=true , address=145.79.227.34:4937 , piece_id=II2LUE2FIDSVNDIBQEF5G2ISPFGL5CFC6NLENMICICJC7HHSPIOA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.160727346 (625h31m18.031810391s) [noise=true , address=145.79.227.33:4889 , piece_id=S34ENDYWNCHS2TLSPVG3WTIBQ2QZ5DV2O3VX76C3IWWEUGZDF2YA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.161214243 (625h31m18.032297288s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:19.631860285 (625h31m19.50294335s) [upload=548124]: Next returned nil segment Dec 31 17:00:19.631881466 (625h31m19.502964501s) [upload=548124]: Waiting for error group managing segments... Dec 31 17:00:20.077955152 (625h31m19.949038197s) [noise=true , address=145.79.227.52:4459 , piece_id=OIZHPVYAWOUFB6EGGW65AYJA7CD2OJHPP5BV7X2BKON6OFSULSRA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.196234912 (625h31m20.067317957s) [noise=true , address=77.90.31.17:4786 , piece_id=C6HSG4XIOQDNTB2RJTJOMFYOMDFDRF4MG2XFAPRSPS726NNBUTLQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.198807942 (625h31m20.069890987s) [noise=true , address=77.90.31.15:4710 , piece_id=AOXLJRW7BEWM7I3VFR64ONRKI3SX5SSUDH47SAXA3IAUPSRPZAOQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.213081430 (625h31m20.084164485s) [noise=true , address=77.90.31.12:4599 , piece_id=QZR3U7P2RGBTEB2LNIOJB6XSRM2NW2KNWPQAB6UR3H6MRRMK7WUQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.285928892 (625h31m20.157011938s) [noise=true , address=145.79.227.40:4072 , piece_id=UE56KE2MGKKROEX4M75BSK4WXD2IH2TDL3DVVFVDPTPX2HRQI4OQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.366975977 (625h31m20.238059022s) [noise=true , address=45.91.116.9:4288 , piece_id=BFB6Z3MDIKZ7AKRLPT6PWQESMDMXHYAWDMN5YCUJXML7WZVAMWOQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.399993577 (625h31m20.271076612s) [noise=true , address=77.90.31.13:4625 , piece_id=DPFGEZR5J2WRRNTISDYXJKWUTZSTRYDY2BS25HTOXQHVRZL52TRA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.977929104 (625h31m20.84901214s) [noise=true , address=145.79.227.44:4204 , piece_id=E55ISFZIJLDZKCIVB7OCVNJZCUJFCWRDKSRDTEZWBLF4FPDNW5BA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.984855463 (625h31m20.855938499s) [noise=true , address=77.90.31.15:4723 , piece_id=CKPWFCY5N3ZCZUBILOS3RZOS6AEFLWN4GH65L25TVYZAXZGQMWVA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.012859246 (625h31m20.883942291s) [noise=true , address=145.79.227.49:4345 , piece_id=RIPCK32N6WLUQYS7EPK6TSEEAQH2DUVO3JZ4VVW3SJSCRCRWZE2A, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.073282851 (625h31m20.944365896s) [noise=true , address=145.79.227.30:4779 , piece_id=HIJQGU3DHFJHVR3ISVSWPZBDX5Y2Q5N4LTUM43GOXCOURUV2AJTQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.132514036 (625h31m21.003597081s) [noise=true , address=145.79.227.49:4343 , piece_id=UFWFBVKIWXCN6QMKXV3D3DCLPEV3DXW27N3J5AK3JWSJ2LXQZAIA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.154431609 (625h31m21.025514664s) [noise=true , address=145.79.227.33:4889 , piece_id=S34ENDYWNCHS2TLSPVG3WTIBQ2QZ5DV2O3VX76C3IWWEUGZDF2YA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.456343772 (625h31m21.327426817s) [noise=true , address=145.79.227.34:4937 , piece_id=II2LUE2FIDSVNDIBQEF5G2ISPFGL5CFC6NLENMICICJC7HHSPIOA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.456381544 (625h31m21.327464589s) [seg_pos={0 0}, upload=548124]: Segment reached optimal threshold of 14 pieces. Dec 31 17:00:21.456811993 (625h31m21.327895059s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection; write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection Dec 31 17:00:21.456824007 (625h31m21.327907052s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457303460 (625h31m21.328386505s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1onAnFbXC1LbUAD31oHGp75v9DpnWmx8LTe6Wazf7XuR3rKnJw): protocol: write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection; write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection Dec 31 17:00:21.457313459 (625h31m21.328396494s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457441663 (625h31m21.328524708s) [noise=true , address=145.79.227.6:4266 , piece_id=GE4XNR6X3DYDYZFUAYZ3X3YVCZ3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wH8jka9Jw8JLYdGKuit8XoJjVLJRDKJ5ePgeei16eax6mk3j): protocol: write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection; write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection Dec 31 17:00:21.457512669 (625h31m21.328595714s) [noise=true , address=145.79.227.6:4266 , piece_id=GE4XNR6X3DYDYZFUAYZ3X3YVCZ3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457641062 (625h31m21.328724107s) [noise=true , address=145.79.227.51:4441 , piece_id=3SEJYHIUPVRHP5QVOWZRB3SBFYVZ7OMRHH3HXQCFGWEZMMCPG5BA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VBayWisMK1n3cwypeiLzGuffonBTzYga7thAZ8pSqrusc7kow): protocol: expected piece hash; context canceled; EOF Dec 31 17:00:21.457649288 (625h31m21.328732333s) [noise=true , address=145.79.227.51:4441 , piece_id=3SEJYHIUPVRHP5QVOWZRB3SBFYVZ7OMRHH3HXQCFGWEZMMCPG5BA, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457667453 (625h31m21.328765977s) [seg_pos={0 0}, upload=548124]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 17:00:21.457773294 (625h31m21.328856339s) [upload=548124]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 17:00:21.457780127 (625h31m21.328863172s) [upload=548124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 17:00:21.754059214 (625h31m21.625142259s) [upload=548124]: Done uploading segments... Dec 31 17:00:21.754081627 (625h31m21.625164662s) [upload=548124]: Upload finished. err: <nil> Duration: 15.426115825s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X....X..X..] duration: 8.166318285s trace: https://jaeger.dev.storj.io/trace/1373dd0e290ba6c5 FAIL (2025-12-31 17:03:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: U6F656GKPNRHZIMAU7EZWP2LGDAPQOW26GQPGH75EKDFOSU6HZMQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: C4HDHTWJEK6HORMQ6WIK7TR4MVJ4GHHHCP56K2IDJG7O4F23GXEQ) context deadline exceeded ======LOGS====== 1 17:00:52.735338143 (625h31m52.606421188s) [upload=548130]: Starting upload Dec 31 17:00:52.735399520 (625h31m52.606482555s) [upload=548130]: Done starting upload Dec 31 17:00:52.735490553 (625h31m52.606573598s) [upload=548130]: Uploading segments... Dec 31 17:00:52.735610542 (625h31m52.606693577s) [upload=548130]: Got next segment. Inline: false Dec 31 17:00:52.735652932 (625h31m52.606735978s) [upload=548130]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 17:00:52.735657832 (625h31m52.606740877s) [upload=548130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 17:00:52.981510289 (625h31m52.852593334s) [seg_pos={0 0}, upload=548130]: Begin upload segment... Dec 31 17:00:52.981556877 (625h31m52.852639922s) [seg_pos={0 0}, upload=548130]: Done begin upload segment. Dec 31 17:00:52.981594789 (625h31m52.852677834s) [noise=true , address=145.79.227.15:4875 , piece_id=DYIJKALM4UAH5GHGX4QD6B3WLTA6F2N3MQLMI5AOEZ7D33ZPNDRA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.981675162 (625h31m52.852758207s) [noise=true , address=145.79.227.32:4835 , piece_id=ALG2N26FSRVDPQN6YOU5DKORV2ZPQRQBVK5XD3FQNRG7XLE2JBUA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.982305923 (625h31m52.853388968s) [noise=true , address=145.79.227.37:4565 , piece_id=O3KYPTHBUNLXDZDQJODLDXH7A5IVQ7G6OWFMUXWAX7YBUSF5MANQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.982731494 (625h31m52.853814539s) [noise=true , address=77.90.31.14:4661 , piece_id=U6F656GKPNRHZIMAU7EZWP2LGDAPQOW26GQPGH75EKDFOSU6HZMQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.983150392 (625h31m52.854233437s) [noise=true , address=45.91.116.12:4227 , piece_id=C4HDHTWJEK6HORMQ6WIK7TR4MVJ4GHHHCP56K2IDJG7O4F23GXEQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.983617190 (625h31m52.854700235s) [noise=true , address=45.91.116.9:4261 , piece_id=67O37BQPRO5VB5Z5BXNRTAHGE7IW4G7NNWFDBRMTZU3I6FS4IWJA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.984033733 (625h31m52.855116778s) [noise=true , address=145.79.227.27:4691 , piece_id=5ICQCX373LFSVDQHKES6QQKIWGZ2LNSH25Y5FEC23MUYFYQNTXUA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.984505772 (625h31m52.855588807s) [noise=true , address=145.79.227.46:4602 , piece_id=4W5SBL3XEV7JYBGP5YZ62TWRDOBYFLJ3ENL3DX44IU4KYSSBQN7A, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.985558736 (625h31m52.856641771s) [noise=true , address=145.79.227.7:4295 , piece_id=SDH27VCHICBL5LULARSOOAHUQ5EVENKGDSPIY3IZYUDHZOIMSQXQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.986391973 (625h31m52.857475008s) [noise=true , address=77.90.31.18:4805 , piece_id=275ZPWN4XS5NE7WBVB7ZYNJRGD6KR7LXKWNTNB7BK3PFAEMSI6HQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.987090583 (625h31m52.858173618s) [noise=true , address=145.79.227.40:4051 , piece_id=QRZONR6L4NW7LVT7BLTXLOQVRZ7KQI67YO5INA4YBULIKZAAXTPA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.988334071 (625h31m52.859417116s) [noise=true , address=77.90.31.21:4877 , piece_id=JPW5YOZODCYKVQSXWGE6ZQHAEN2HZH5TNCRRD76C6VEXEFR47FQA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.991339614 (625h31m52.862422659s) [noise=true , address=145.79.227.24:4583 , piece_id=WE5VKQ2HATUM6NRZJT3PSRHFZP74UFRS23SJA6LLL63VBCQRRMSQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.993748692 (625h31m52.864831747s) [noise=true , address=145.79.227.10:4401 , piece_id=6LT37OSUXSWCDPUEZUUYD2BPOTRQVPLJZSJPYXMAXTHXMJQ6CLWQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.993776514 (625h31m52.864859559s) [noise=true , address=145.79.227.13:4804 , piece_id=2RBUZRQIB2HW3OMTZO2BDPMCKRADGAODX2GOPR2LAC7WFJ3SZSTA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.995322709 (625h31m52.866405754s) [noise=true , address=145.79.227.6:4259 , piece_id=4YAYCJAFK6YO6BY6LS3XEKN3NWMCCGX7RMZKFLFPSGDGF5SVTPKA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:54.586869504 (625h31m54.457952549s) [upload=548130]: Next returned nil segment Dec 31 17:00:54.586887639 (625h31m54.457970684s) [upload=548130]: Waiting for error group managing segments... Dec 31 17:00:54.969318389 (625h31m54.840401434s) [noise=true , address=145.79.227.37:4565 , piece_id=O3KYPTHBUNLXDZDQJODLDXH7A5IVQ7G6OWFMUXWAX7YBUSF5MANQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.970514227 (625h31m54.841597272s) [noise=true , address=145.79.227.32:4835 , piece_id=ALG2N26FSRVDPQN6YOU5DKORV2ZPQRQBVK5XD3FQNRG7XLE2JBUA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.976748708 (625h31m54.847831754s) [noise=true , address=145.79.227.7:4295 , piece_id=SDH27VCHICBL5LULARSOOAHUQ5EVENKGDSPIY3IZYUDHZOIMSQXQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.979387142 (625h31m54.850470177s) [noise=true , address=145.79.227.15:4875 , piece_id=DYIJKALM4UAH5GHGX4QD6B3WLTA6F2N3MQLMI5AOEZ7D33ZPNDRA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.994174269 (625h31m54.865257314s) [noise=true , address=145.79.227.13:4804 , piece_id=2RBUZRQIB2HW3OMTZO2BDPMCKRADGAODX2GOPR2LAC7WFJ3SZSTA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.996871525 (625h31m54.86795457s) [noise=true , address=145.79.227.46:4602 , piece_id=4W5SBL3XEV7JYBGP5YZ62TWRDOBYFLJ3ENL3DX44IU4KYSSBQN7A, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.003898666 (625h31m54.874981731s) [noise=true , address=145.79.227.27:4691 , piece_id=5ICQCX373LFSVDQHKES6QQKIWGZ2LNSH25Y5FEC23MUYFYQNTXUA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.005164556 (625h31m54.876247611s) [noise=true , address=145.79.227.6:4259 , piece_id=4YAYCJAFK6YO6BY6LS3XEKN3NWMCCGX7RMZKFLFPSGDGF5SVTPKA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.005408150 (625h31m54.876491215s) [noise=true , address=145.79.227.24:4583 , piece_id=WE5VKQ2HATUM6NRZJT3PSRHFZP74UFRS23SJA6LLL63VBCQRRMSQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.010608472 (625h31m54.881691507s) [noise=true , address=145.79.227.10:4401 , piece_id=6LT37OSUXSWCDPUEZUUYD2BPOTRQVPLJZSJPYXMAXTHXMJQ6CLWQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.168517623 (625h31m55.039600668s) [noise=true , address=77.90.31.14:4661 , piece_id=U6F656GKPNRHZIMAU7EZWP2LGDAPQOW26GQPGH75EKDFOSU6HZMQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.319279103 (625h31m55.190362148s) [noise=true , address=45.91.116.12:4227 , piece_id=C4HDHTWJEK6HORMQ6WIK7TR4MVJ4GHHHCP56K2IDJG7O4F23GXEQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.518274919 (625h31m55.389357964s) [noise=true , address=45.91.116.9:4261 , piece_id=67O37BQPRO5VB5Z5BXNRTAHGE7IW4G7NNWFDBRMTZU3I6FS4IWJA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.518335765 (625h31m55.3894188s) [seg_pos={0 0}, upload=548130]: Segment reached optimal threshold of 13 pieces. Dec 31 17:00:55.518502642 (625h31m55.389585687s) [noise=true , address=145.79.227.40:4051 , piece_id=QRZONR6L4NW7LVT7BLTXLOQVRZ7KQI67YO5INA4YBULIKZAAXTPA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 17:00:55.518508594 (625h31m55.389591639s) [noise=true , address=145.79.227.40:4051 , piece_id=QRZONR6L4NW7LVT7BLTXLOQVRZ7KQI67YO5INA4YBULIKZAAXTPA, seg_pos={0 0}, upload=548130]: Long tail cancellation due to optimal threshold Dec 31 17:00:55.518554912 (625h31m55.389637947s) [noise=true , address=77.90.31.18:4805 , piece_id=275ZPWN4XS5NE7WBVB7ZYNJRGD6KR7LXKWNTNB7BK3PFAEMSI6HQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 31 17:00:55.518565051 (625h31m55.389648086s) [noise=true , address=77.90.31.18:4805 , piece_id=275ZPWN4XS5NE7WBVB7ZYNJRGD6KR7LXKWNTNB7BK3PFAEMSI6HQ, seg_pos={0 0}, upload=548130]: Long tail cancellation due to optimal threshold Dec 31 17:00:55.518715357 (625h31m55.389798402s) [noise=true , address=77.90.31.21:4877 , piece_id=JPW5YOZODCYKVQSXWGE6ZQHAEN2HZH5TNCRRD76C6VEXEFR47FQA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46340->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:46340->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46340->77.90.31.21:4877: use of closed network connection Dec 31 17:00:55.518722210 (625h31m55.389805245s) [noise=true , address=77.90.31.21:4877 , piece_id=JPW5YOZODCYKVQSXWGE6ZQHAEN2HZH5TNCRRD76C6VEXEFR47FQA, seg_pos={0 0}, upload=548130]: Long tail cancellation due to optimal threshold Dec 31 17:00:55.518746115 (625h31m55.38982915s) [seg_pos={0 0}, upload=548130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 17:00:55.518834514 (625h31m55.389917549s) [upload=548130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 17:00:55.518839984 (625h31m55.389923029s) [upload=548130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 17:00:55.813662567 (625h31m55.684745622s) [upload=548130]: Done uploading segments... Dec 31 17:00:55.813687064 (625h31m55.684770119s) [upload=548130]: Upload finished. err: <nil> Duration: 8.166350667s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.XXX...X..] duration: 3.391624316s trace: https://jaeger.dev.storj.io/trace/71adf0745edb131f FAIL (2025-12-31 17:02:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.391652229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.....................................X......X...X..] duration: 18.801492609s trace: https://jaeger.dev.storj.io/trace/70b0cb39dd42d67b FAIL (2025-12-31 17:02:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.156432368 (625h31m18.027515413s) [noise=true , address=77.90.31.15:4710 , piece_id=AOXLJRW7BEWM7I3VFR64ONRKI3SX5SSUDH47SAXA3IAUPSRPZAOQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157072997 (625h31m18.028156042s) [noise=true , address=145.79.227.52:4459 , piece_id=OIZHPVYAWOUFB6EGGW65AYJA7CD2OJHPP5BV7X2BKON6OFSULSRA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157213074 (625h31m18.028296159s) [noise=true , address=77.90.31.15:4723 , piece_id=CKPWFCY5N3ZCZUBILOS3RZOS6AEFLWN4GH65L25TVYZAXZGQMWVA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157552671 (625h31m18.028635716s) [noise=true , address=77.90.31.17:4786 , piece_id=C6HSG4XIOQDNTB2RJTJOMFYOMDFDRF4MG2XFAPRSPS726NNBUTLQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157787158 (625h31m18.028870203s) [noise=true , address=145.79.227.40:4072 , piece_id=UE56KE2MGKKROEX4M75BSK4WXD2IH2TDL3DVVFVDPTPX2HRQI4OQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158098741 (625h31m18.029181786s) [noise=true , address=77.90.31.13:4625 , piece_id=DPFGEZR5J2WRRNTISDYXJKWUTZSTRYDY2BS25HTOXQHVRZL52TRA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158386689 (625h31m18.029469734s) [noise=true , address=145.79.227.49:4343 , piece_id=UFWFBVKIWXCN6QMKXV3D3DCLPEV3DXW27N3J5AK3JWSJ2LXQZAIA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158597901 (625h31m18.029680946s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.159008944 (625h31m18.030091989s) [noise=true , address=145.79.227.30:4779 , piece_id=HIJQGU3DHFJHVR3ISVSWPZBDX5Y2Q5N4LTUM43GOXCOURUV2AJTQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.159761006 (625h31m18.030844051s) [noise=true , address=145.79.227.49:4345 , piece_id=RIPCK32N6WLUQYS7EPK6TSEEAQH2DUVO3JZ4VVW3SJSCRCRWZE2A, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.160164584 (625h31m18.03124763s) [noise=true , address=145.79.227.34:4937 , piece_id=II2LUE2FIDSVNDIBQEF5G2ISPFGL5CFC6NLENMICICJC7HHSPIOA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.160727346 (625h31m18.031810391s) [noise=true , address=145.79.227.33:4889 , piece_id=S34ENDYWNCHS2TLSPVG3WTIBQ2QZ5DV2O3VX76C3IWWEUGZDF2YA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.161214243 (625h31m18.032297288s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:19.631860285 (625h31m19.50294335s) [upload=548124]: Next returned nil segment Dec 31 17:00:19.631881466 (625h31m19.502964501s) [upload=548124]: Waiting for error group managing segments... Dec 31 17:00:20.077955152 (625h31m19.949038197s) [noise=true , address=145.79.227.52:4459 , piece_id=OIZHPVYAWOUFB6EGGW65AYJA7CD2OJHPP5BV7X2BKON6OFSULSRA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.196234912 (625h31m20.067317957s) [noise=true , address=77.90.31.17:4786 , piece_id=C6HSG4XIOQDNTB2RJTJOMFYOMDFDRF4MG2XFAPRSPS726NNBUTLQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.198807942 (625h31m20.069890987s) [noise=true , address=77.90.31.15:4710 , piece_id=AOXLJRW7BEWM7I3VFR64ONRKI3SX5SSUDH47SAXA3IAUPSRPZAOQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.213081430 (625h31m20.084164485s) [noise=true , address=77.90.31.12:4599 , piece_id=QZR3U7P2RGBTEB2LNIOJB6XSRM2NW2KNWPQAB6UR3H6MRRMK7WUQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.285928892 (625h31m20.157011938s) [noise=true , address=145.79.227.40:4072 , piece_id=UE56KE2MGKKROEX4M75BSK4WXD2IH2TDL3DVVFVDPTPX2HRQI4OQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.366975977 (625h31m20.238059022s) [noise=true , address=45.91.116.9:4288 , piece_id=BFB6Z3MDIKZ7AKRLPT6PWQESMDMXHYAWDMN5YCUJXML7WZVAMWOQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.399993577 (625h31m20.271076612s) [noise=true , address=77.90.31.13:4625 , piece_id=DPFGEZR5J2WRRNTISDYXJKWUTZSTRYDY2BS25HTOXQHVRZL52TRA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.977929104 (625h31m20.84901214s) [noise=true , address=145.79.227.44:4204 , piece_id=E55ISFZIJLDZKCIVB7OCVNJZCUJFCWRDKSRDTEZWBLF4FPDNW5BA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.984855463 (625h31m20.855938499s) [noise=true , address=77.90.31.15:4723 , piece_id=CKPWFCY5N3ZCZUBILOS3RZOS6AEFLWN4GH65L25TVYZAXZGQMWVA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.012859246 (625h31m20.883942291s) [noise=true , address=145.79.227.49:4345 , piece_id=RIPCK32N6WLUQYS7EPK6TSEEAQH2DUVO3JZ4VVW3SJSCRCRWZE2A, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.073282851 (625h31m20.944365896s) [noise=true , address=145.79.227.30:4779 , piece_id=HIJQGU3DHFJHVR3ISVSWPZBDX5Y2Q5N4LTUM43GOXCOURUV2AJTQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.132514036 (625h31m21.003597081s) [noise=true , address=145.79.227.49:4343 , piece_id=UFWFBVKIWXCN6QMKXV3D3DCLPEV3DXW27N3J5AK3JWSJ2LXQZAIA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.154431609 (625h31m21.025514664s) [noise=true , address=145.79.227.33:4889 , piece_id=S34ENDYWNCHS2TLSPVG3WTIBQ2QZ5DV2O3VX76C3IWWEUGZDF2YA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.456343772 (625h31m21.327426817s) [noise=true , address=145.79.227.34:4937 , piece_id=II2LUE2FIDSVNDIBQEF5G2ISPFGL5CFC6NLENMICICJC7HHSPIOA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.456381544 (625h31m21.327464589s) [seg_pos={0 0}, upload=548124]: Segment reached optimal threshold of 14 pieces. Dec 31 17:00:21.456811993 (625h31m21.327895059s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection; write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection Dec 31 17:00:21.456824007 (625h31m21.327907052s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457303460 (625h31m21.328386505s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1onAnFbXC1LbUAD31oHGp75v9DpnWmx8LTe6Wazf7XuR3rKnJw): protocol: write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection; write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection Dec 31 17:00:21.457313459 (625h31m21.328396494s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457441663 (625h31m21.328524708s) [noise=true , address=145.79.227.6:4266 , piece_id=GE4XNR6X3DYDYZFUAYZ3X3YVCZ3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wH8jka9Jw8JLYdGKuit8XoJjVLJRDKJ5ePgeei16eax6mk3j): protocol: write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection; write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection Dec 31 17:00:21.457512669 (625h31m21.328595714s) [noise=true , address=145.79.227.6:4266 , piece_id=GE4XNR6X3DYDYZFUAYZ3X3YVCZ3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457641062 (625h31m21.328724107s) [noise=true , address=145.79.227.51:4441 , piece_id=3SEJYHIUPVRHP5QVOWZRB3SBFYVZ7OMRHH3HXQCFGWEZMMCPG5BA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VBayWisMK1n3cwypeiLzGuffonBTzYga7thAZ8pSqrusc7kow): protocol: expected piece hash; context canceled; EOF Dec 31 17:00:21.457649288 (625h31m21.328732333s) [noise=true , address=145.79.227.51:4441 , piece_id=3SEJYHIUPVRHP5QVOWZRB3SBFYVZ7OMRHH3HXQCFGWEZMMCPG5BA, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457667453 (625h31m21.328765977s) [seg_pos={0 0}, upload=548124]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 17:00:21.457773294 (625h31m21.328856339s) [upload=548124]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 17:00:21.457780127 (625h31m21.328863172s) [upload=548124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 17:00:21.754059214 (625h31m21.625142259s) [upload=548124]: Done uploading segments... Dec 31 17:00:21.754081627 (625h31m21.625164662s) [upload=548124]: Upload finished. err: <nil> Duration: 18.801510553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X....X..X.] duration: 6.182492556s trace: https://jaeger.dev.storj.io/trace/1373dd0e290ba6c5 FAIL (2025-12-31 17:02:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: U6F656GKPNRHZIMAU7EZWP2LGDAPQOW26GQPGH75EKDFOSU6HZMQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: C4HDHTWJEK6HORMQ6WIK7TR4MVJ4GHHHCP56K2IDJG7O4F23GXEQ) context deadline exceeded ======LOGS====== 1 17:00:52.735338143 (625h31m52.606421188s) [upload=548130]: Starting upload Dec 31 17:00:52.735399520 (625h31m52.606482555s) [upload=548130]: Done starting upload Dec 31 17:00:52.735490553 (625h31m52.606573598s) [upload=548130]: Uploading segments... Dec 31 17:00:52.735610542 (625h31m52.606693577s) [upload=548130]: Got next segment. Inline: false Dec 31 17:00:52.735652932 (625h31m52.606735978s) [upload=548130]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 17:00:52.735657832 (625h31m52.606740877s) [upload=548130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 17:00:52.981510289 (625h31m52.852593334s) [seg_pos={0 0}, upload=548130]: Begin upload segment... Dec 31 17:00:52.981556877 (625h31m52.852639922s) [seg_pos={0 0}, upload=548130]: Done begin upload segment. Dec 31 17:00:52.981594789 (625h31m52.852677834s) [noise=true , address=145.79.227.15:4875 , piece_id=DYIJKALM4UAH5GHGX4QD6B3WLTA6F2N3MQLMI5AOEZ7D33ZPNDRA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.981675162 (625h31m52.852758207s) [noise=true , address=145.79.227.32:4835 , piece_id=ALG2N26FSRVDPQN6YOU5DKORV2ZPQRQBVK5XD3FQNRG7XLE2JBUA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.982305923 (625h31m52.853388968s) [noise=true , address=145.79.227.37:4565 , piece_id=O3KYPTHBUNLXDZDQJODLDXH7A5IVQ7G6OWFMUXWAX7YBUSF5MANQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.982731494 (625h31m52.853814539s) [noise=true , address=77.90.31.14:4661 , piece_id=U6F656GKPNRHZIMAU7EZWP2LGDAPQOW26GQPGH75EKDFOSU6HZMQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.983150392 (625h31m52.854233437s) [noise=true , address=45.91.116.12:4227 , piece_id=C4HDHTWJEK6HORMQ6WIK7TR4MVJ4GHHHCP56K2IDJG7O4F23GXEQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.983617190 (625h31m52.854700235s) [noise=true , address=45.91.116.9:4261 , piece_id=67O37BQPRO5VB5Z5BXNRTAHGE7IW4G7NNWFDBRMTZU3I6FS4IWJA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.984033733 (625h31m52.855116778s) [noise=true , address=145.79.227.27:4691 , piece_id=5ICQCX373LFSVDQHKES6QQKIWGZ2LNSH25Y5FEC23MUYFYQNTXUA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.984505772 (625h31m52.855588807s) [noise=true , address=145.79.227.46:4602 , piece_id=4W5SBL3XEV7JYBGP5YZ62TWRDOBYFLJ3ENL3DX44IU4KYSSBQN7A, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.985558736 (625h31m52.856641771s) [noise=true , address=145.79.227.7:4295 , piece_id=SDH27VCHICBL5LULARSOOAHUQ5EVENKGDSPIY3IZYUDHZOIMSQXQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.986391973 (625h31m52.857475008s) [noise=true , address=77.90.31.18:4805 , piece_id=275ZPWN4XS5NE7WBVB7ZYNJRGD6KR7LXKWNTNB7BK3PFAEMSI6HQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.987090583 (625h31m52.858173618s) [noise=true , address=145.79.227.40:4051 , piece_id=QRZONR6L4NW7LVT7BLTXLOQVRZ7KQI67YO5INA4YBULIKZAAXTPA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.988334071 (625h31m52.859417116s) [noise=true , address=77.90.31.21:4877 , piece_id=JPW5YOZODCYKVQSXWGE6ZQHAEN2HZH5TNCRRD76C6VEXEFR47FQA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.991339614 (625h31m52.862422659s) [noise=true , address=145.79.227.24:4583 , piece_id=WE5VKQ2HATUM6NRZJT3PSRHFZP74UFRS23SJA6LLL63VBCQRRMSQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.993748692 (625h31m52.864831747s) [noise=true , address=145.79.227.10:4401 , piece_id=6LT37OSUXSWCDPUEZUUYD2BPOTRQVPLJZSJPYXMAXTHXMJQ6CLWQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.993776514 (625h31m52.864859559s) [noise=true , address=145.79.227.13:4804 , piece_id=2RBUZRQIB2HW3OMTZO2BDPMCKRADGAODX2GOPR2LAC7WFJ3SZSTA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.995322709 (625h31m52.866405754s) [noise=true , address=145.79.227.6:4259 , piece_id=4YAYCJAFK6YO6BY6LS3XEKN3NWMCCGX7RMZKFLFPSGDGF5SVTPKA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:54.586869504 (625h31m54.457952549s) [upload=548130]: Next returned nil segment Dec 31 17:00:54.586887639 (625h31m54.457970684s) [upload=548130]: Waiting for error group managing segments... Dec 31 17:00:54.969318389 (625h31m54.840401434s) [noise=true , address=145.79.227.37:4565 , piece_id=O3KYPTHBUNLXDZDQJODLDXH7A5IVQ7G6OWFMUXWAX7YBUSF5MANQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.970514227 (625h31m54.841597272s) [noise=true , address=145.79.227.32:4835 , piece_id=ALG2N26FSRVDPQN6YOU5DKORV2ZPQRQBVK5XD3FQNRG7XLE2JBUA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.976748708 (625h31m54.847831754s) [noise=true , address=145.79.227.7:4295 , piece_id=SDH27VCHICBL5LULARSOOAHUQ5EVENKGDSPIY3IZYUDHZOIMSQXQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.979387142 (625h31m54.850470177s) [noise=true , address=145.79.227.15:4875 , piece_id=DYIJKALM4UAH5GHGX4QD6B3WLTA6F2N3MQLMI5AOEZ7D33ZPNDRA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.994174269 (625h31m54.865257314s) [noise=true , address=145.79.227.13:4804 , piece_id=2RBUZRQIB2HW3OMTZO2BDPMCKRADGAODX2GOPR2LAC7WFJ3SZSTA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.996871525 (625h31m54.86795457s) [noise=true , address=145.79.227.46:4602 , piece_id=4W5SBL3XEV7JYBGP5YZ62TWRDOBYFLJ3ENL3DX44IU4KYSSBQN7A, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.003898666 (625h31m54.874981731s) [noise=true , address=145.79.227.27:4691 , piece_id=5ICQCX373LFSVDQHKES6QQKIWGZ2LNSH25Y5FEC23MUYFYQNTXUA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.005164556 (625h31m54.876247611s) [noise=true , address=145.79.227.6:4259 , piece_id=4YAYCJAFK6YO6BY6LS3XEKN3NWMCCGX7RMZKFLFPSGDGF5SVTPKA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.005408150 (625h31m54.876491215s) [noise=true , address=145.79.227.24:4583 , piece_id=WE5VKQ2HATUM6NRZJT3PSRHFZP74UFRS23SJA6LLL63VBCQRRMSQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.010608472 (625h31m54.881691507s) [noise=true , address=145.79.227.10:4401 , piece_id=6LT37OSUXSWCDPUEZUUYD2BPOTRQVPLJZSJPYXMAXTHXMJQ6CLWQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.168517623 (625h31m55.039600668s) [noise=true , address=77.90.31.14:4661 , piece_id=U6F656GKPNRHZIMAU7EZWP2LGDAPQOW26GQPGH75EKDFOSU6HZMQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.319279103 (625h31m55.190362148s) [noise=true , address=45.91.116.12:4227 , piece_id=C4HDHTWJEK6HORMQ6WIK7TR4MVJ4GHHHCP56K2IDJG7O4F23GXEQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.518274919 (625h31m55.389357964s) [noise=true , address=45.91.116.9:4261 , piece_id=67O37BQPRO5VB5Z5BXNRTAHGE7IW4G7NNWFDBRMTZU3I6FS4IWJA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.518335765 (625h31m55.3894188s) [seg_pos={0 0}, upload=548130]: Segment reached optimal threshold of 13 pieces. Dec 31 17:00:55.518502642 (625h31m55.389585687s) [noise=true , address=145.79.227.40:4051 , piece_id=QRZONR6L4NW7LVT7BLTXLOQVRZ7KQI67YO5INA4YBULIKZAAXTPA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 17:00:55.518508594 (625h31m55.389591639s) [noise=true , address=145.79.227.40:4051 , piece_id=QRZONR6L4NW7LVT7BLTXLOQVRZ7KQI67YO5INA4YBULIKZAAXTPA, seg_pos={0 0}, upload=548130]: Long tail cancellation due to optimal threshold Dec 31 17:00:55.518554912 (625h31m55.389637947s) [noise=true , address=77.90.31.18:4805 , piece_id=275ZPWN4XS5NE7WBVB7ZYNJRGD6KR7LXKWNTNB7BK3PFAEMSI6HQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 31 17:00:55.518565051 (625h31m55.389648086s) [noise=true , address=77.90.31.18:4805 , piece_id=275ZPWN4XS5NE7WBVB7ZYNJRGD6KR7LXKWNTNB7BK3PFAEMSI6HQ, seg_pos={0 0}, upload=548130]: Long tail cancellation due to optimal threshold Dec 31 17:00:55.518715357 (625h31m55.389798402s) [noise=true , address=77.90.31.21:4877 , piece_id=JPW5YOZODCYKVQSXWGE6ZQHAEN2HZH5TNCRRD76C6VEXEFR47FQA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46340->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:46340->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46340->77.90.31.21:4877: use of closed network connection Dec 31 17:00:55.518722210 (625h31m55.389805245s) [noise=true , address=77.90.31.21:4877 , piece_id=JPW5YOZODCYKVQSXWGE6ZQHAEN2HZH5TNCRRD76C6VEXEFR47FQA, seg_pos={0 0}, upload=548130]: Long tail cancellation due to optimal threshold Dec 31 17:00:55.518746115 (625h31m55.38982915s) [seg_pos={0 0}, upload=548130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 17:00:55.518834514 (625h31m55.389917549s) [upload=548130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 17:00:55.518839984 (625h31m55.389923029s) [upload=548130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 17:00:55.813662567 (625h31m55.684745622s) [upload=548130]: Done uploading segments... Dec 31 17:00:55.813687064 (625h31m55.684770119s) [upload=548130]: Upload finished. err: <nil> Duration: 6.182525398s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.....................................X......X...X.] duration: 38.068807338s trace: https://jaeger.dev.storj.io/trace/70b0cb39dd42d67b FAIL (2025-12-31 17:01:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.156432368 (625h31m18.027515413s) [noise=true , address=77.90.31.15:4710 , piece_id=AOXLJRW7BEWM7I3VFR64ONRKI3SX5SSUDH47SAXA3IAUPSRPZAOQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157072997 (625h31m18.028156042s) [noise=true , address=145.79.227.52:4459 , piece_id=OIZHPVYAWOUFB6EGGW65AYJA7CD2OJHPP5BV7X2BKON6OFSULSRA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157213074 (625h31m18.028296159s) [noise=true , address=77.90.31.15:4723 , piece_id=CKPWFCY5N3ZCZUBILOS3RZOS6AEFLWN4GH65L25TVYZAXZGQMWVA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157552671 (625h31m18.028635716s) [noise=true , address=77.90.31.17:4786 , piece_id=C6HSG4XIOQDNTB2RJTJOMFYOMDFDRF4MG2XFAPRSPS726NNBUTLQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157787158 (625h31m18.028870203s) [noise=true , address=145.79.227.40:4072 , piece_id=UE56KE2MGKKROEX4M75BSK4WXD2IH2TDL3DVVFVDPTPX2HRQI4OQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158098741 (625h31m18.029181786s) [noise=true , address=77.90.31.13:4625 , piece_id=DPFGEZR5J2WRRNTISDYXJKWUTZSTRYDY2BS25HTOXQHVRZL52TRA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158386689 (625h31m18.029469734s) [noise=true , address=145.79.227.49:4343 , piece_id=UFWFBVKIWXCN6QMKXV3D3DCLPEV3DXW27N3J5AK3JWSJ2LXQZAIA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158597901 (625h31m18.029680946s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.159008944 (625h31m18.030091989s) [noise=true , address=145.79.227.30:4779 , piece_id=HIJQGU3DHFJHVR3ISVSWPZBDX5Y2Q5N4LTUM43GOXCOURUV2AJTQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.159761006 (625h31m18.030844051s) [noise=true , address=145.79.227.49:4345 , piece_id=RIPCK32N6WLUQYS7EPK6TSEEAQH2DUVO3JZ4VVW3SJSCRCRWZE2A, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.160164584 (625h31m18.03124763s) [noise=true , address=145.79.227.34:4937 , piece_id=II2LUE2FIDSVNDIBQEF5G2ISPFGL5CFC6NLENMICICJC7HHSPIOA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.160727346 (625h31m18.031810391s) [noise=true , address=145.79.227.33:4889 , piece_id=S34ENDYWNCHS2TLSPVG3WTIBQ2QZ5DV2O3VX76C3IWWEUGZDF2YA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.161214243 (625h31m18.032297288s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:19.631860285 (625h31m19.50294335s) [upload=548124]: Next returned nil segment Dec 31 17:00:19.631881466 (625h31m19.502964501s) [upload=548124]: Waiting for error group managing segments... Dec 31 17:00:20.077955152 (625h31m19.949038197s) [noise=true , address=145.79.227.52:4459 , piece_id=OIZHPVYAWOUFB6EGGW65AYJA7CD2OJHPP5BV7X2BKON6OFSULSRA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.196234912 (625h31m20.067317957s) [noise=true , address=77.90.31.17:4786 , piece_id=C6HSG4XIOQDNTB2RJTJOMFYOMDFDRF4MG2XFAPRSPS726NNBUTLQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.198807942 (625h31m20.069890987s) [noise=true , address=77.90.31.15:4710 , piece_id=AOXLJRW7BEWM7I3VFR64ONRKI3SX5SSUDH47SAXA3IAUPSRPZAOQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.213081430 (625h31m20.084164485s) [noise=true , address=77.90.31.12:4599 , piece_id=QZR3U7P2RGBTEB2LNIOJB6XSRM2NW2KNWPQAB6UR3H6MRRMK7WUQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.285928892 (625h31m20.157011938s) [noise=true , address=145.79.227.40:4072 , piece_id=UE56KE2MGKKROEX4M75BSK4WXD2IH2TDL3DVVFVDPTPX2HRQI4OQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.366975977 (625h31m20.238059022s) [noise=true , address=45.91.116.9:4288 , piece_id=BFB6Z3MDIKZ7AKRLPT6PWQESMDMXHYAWDMN5YCUJXML7WZVAMWOQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.399993577 (625h31m20.271076612s) [noise=true , address=77.90.31.13:4625 , piece_id=DPFGEZR5J2WRRNTISDYXJKWUTZSTRYDY2BS25HTOXQHVRZL52TRA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.977929104 (625h31m20.84901214s) [noise=true , address=145.79.227.44:4204 , piece_id=E55ISFZIJLDZKCIVB7OCVNJZCUJFCWRDKSRDTEZWBLF4FPDNW5BA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.984855463 (625h31m20.855938499s) [noise=true , address=77.90.31.15:4723 , piece_id=CKPWFCY5N3ZCZUBILOS3RZOS6AEFLWN4GH65L25TVYZAXZGQMWVA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.012859246 (625h31m20.883942291s) [noise=true , address=145.79.227.49:4345 , piece_id=RIPCK32N6WLUQYS7EPK6TSEEAQH2DUVO3JZ4VVW3SJSCRCRWZE2A, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.073282851 (625h31m20.944365896s) [noise=true , address=145.79.227.30:4779 , piece_id=HIJQGU3DHFJHVR3ISVSWPZBDX5Y2Q5N4LTUM43GOXCOURUV2AJTQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.132514036 (625h31m21.003597081s) [noise=true , address=145.79.227.49:4343 , piece_id=UFWFBVKIWXCN6QMKXV3D3DCLPEV3DXW27N3J5AK3JWSJ2LXQZAIA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.154431609 (625h31m21.025514664s) [noise=true , address=145.79.227.33:4889 , piece_id=S34ENDYWNCHS2TLSPVG3WTIBQ2QZ5DV2O3VX76C3IWWEUGZDF2YA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.456343772 (625h31m21.327426817s) [noise=true , address=145.79.227.34:4937 , piece_id=II2LUE2FIDSVNDIBQEF5G2ISPFGL5CFC6NLENMICICJC7HHSPIOA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.456381544 (625h31m21.327464589s) [seg_pos={0 0}, upload=548124]: Segment reached optimal threshold of 14 pieces. Dec 31 17:00:21.456811993 (625h31m21.327895059s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection; write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection Dec 31 17:00:21.456824007 (625h31m21.327907052s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457303460 (625h31m21.328386505s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1onAnFbXC1LbUAD31oHGp75v9DpnWmx8LTe6Wazf7XuR3rKnJw): protocol: write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection; write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection Dec 31 17:00:21.457313459 (625h31m21.328396494s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457441663 (625h31m21.328524708s) [noise=true , address=145.79.227.6:4266 , piece_id=GE4XNR6X3DYDYZFUAYZ3X3YVCZ3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wH8jka9Jw8JLYdGKuit8XoJjVLJRDKJ5ePgeei16eax6mk3j): protocol: write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection; write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection Dec 31 17:00:21.457512669 (625h31m21.328595714s) [noise=true , address=145.79.227.6:4266 , piece_id=GE4XNR6X3DYDYZFUAYZ3X3YVCZ3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457641062 (625h31m21.328724107s) [noise=true , address=145.79.227.51:4441 , piece_id=3SEJYHIUPVRHP5QVOWZRB3SBFYVZ7OMRHH3HXQCFGWEZMMCPG5BA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VBayWisMK1n3cwypeiLzGuffonBTzYga7thAZ8pSqrusc7kow): protocol: expected piece hash; context canceled; EOF Dec 31 17:00:21.457649288 (625h31m21.328732333s) [noise=true , address=145.79.227.51:4441 , piece_id=3SEJYHIUPVRHP5QVOWZRB3SBFYVZ7OMRHH3HXQCFGWEZMMCPG5BA, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457667453 (625h31m21.328765977s) [seg_pos={0 0}, upload=548124]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 17:00:21.457773294 (625h31m21.328856339s) [upload=548124]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 17:00:21.457780127 (625h31m21.328863172s) [upload=548124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 17:00:21.754059214 (625h31m21.625142259s) [upload=548124]: Done uploading segments... Dec 31 17:00:21.754081627 (625h31m21.625164662s) [upload=548124]: Upload finished. err: <nil> Duration: 38.068826203s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.XXX...X.] duration: 3.408099735s trace: https://jaeger.dev.storj.io/trace/71adf0745edb131f FAIL (2025-12-31 17:01:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.408131214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X....X..X] duration: 30.003094961s trace: https://jaeger.dev.storj.io/trace/1373dd0e290ba6c5 FAIL (2025-12-31 17:01:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: U6F656GKPNRHZIMAU7EZWP2LGDAPQOW26GQPGH75EKDFOSU6HZMQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: C4HDHTWJEK6HORMQ6WIK7TR4MVJ4GHHHCP56K2IDJG7O4F23GXEQ) context deadline exceeded ======LOGS====== 1 17:00:52.735338143 (625h31m52.606421188s) [upload=548130]: Starting upload Dec 31 17:00:52.735399520 (625h31m52.606482555s) [upload=548130]: Done starting upload Dec 31 17:00:52.735490553 (625h31m52.606573598s) [upload=548130]: Uploading segments... Dec 31 17:00:52.735610542 (625h31m52.606693577s) [upload=548130]: Got next segment. Inline: false Dec 31 17:00:52.735652932 (625h31m52.606735978s) [upload=548130]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 17:00:52.735657832 (625h31m52.606740877s) [upload=548130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 17:00:52.981510289 (625h31m52.852593334s) [seg_pos={0 0}, upload=548130]: Begin upload segment... Dec 31 17:00:52.981556877 (625h31m52.852639922s) [seg_pos={0 0}, upload=548130]: Done begin upload segment. Dec 31 17:00:52.981594789 (625h31m52.852677834s) [noise=true , address=145.79.227.15:4875 , piece_id=DYIJKALM4UAH5GHGX4QD6B3WLTA6F2N3MQLMI5AOEZ7D33ZPNDRA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.981675162 (625h31m52.852758207s) [noise=true , address=145.79.227.32:4835 , piece_id=ALG2N26FSRVDPQN6YOU5DKORV2ZPQRQBVK5XD3FQNRG7XLE2JBUA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.982305923 (625h31m52.853388968s) [noise=true , address=145.79.227.37:4565 , piece_id=O3KYPTHBUNLXDZDQJODLDXH7A5IVQ7G6OWFMUXWAX7YBUSF5MANQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.982731494 (625h31m52.853814539s) [noise=true , address=77.90.31.14:4661 , piece_id=U6F656GKPNRHZIMAU7EZWP2LGDAPQOW26GQPGH75EKDFOSU6HZMQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.983150392 (625h31m52.854233437s) [noise=true , address=45.91.116.12:4227 , piece_id=C4HDHTWJEK6HORMQ6WIK7TR4MVJ4GHHHCP56K2IDJG7O4F23GXEQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.983617190 (625h31m52.854700235s) [noise=true , address=45.91.116.9:4261 , piece_id=67O37BQPRO5VB5Z5BXNRTAHGE7IW4G7NNWFDBRMTZU3I6FS4IWJA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.984033733 (625h31m52.855116778s) [noise=true , address=145.79.227.27:4691 , piece_id=5ICQCX373LFSVDQHKES6QQKIWGZ2LNSH25Y5FEC23MUYFYQNTXUA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.984505772 (625h31m52.855588807s) [noise=true , address=145.79.227.46:4602 , piece_id=4W5SBL3XEV7JYBGP5YZ62TWRDOBYFLJ3ENL3DX44IU4KYSSBQN7A, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.985558736 (625h31m52.856641771s) [noise=true , address=145.79.227.7:4295 , piece_id=SDH27VCHICBL5LULARSOOAHUQ5EVENKGDSPIY3IZYUDHZOIMSQXQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.986391973 (625h31m52.857475008s) [noise=true , address=77.90.31.18:4805 , piece_id=275ZPWN4XS5NE7WBVB7ZYNJRGD6KR7LXKWNTNB7BK3PFAEMSI6HQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.987090583 (625h31m52.858173618s) [noise=true , address=145.79.227.40:4051 , piece_id=QRZONR6L4NW7LVT7BLTXLOQVRZ7KQI67YO5INA4YBULIKZAAXTPA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.988334071 (625h31m52.859417116s) [noise=true , address=77.90.31.21:4877 , piece_id=JPW5YOZODCYKVQSXWGE6ZQHAEN2HZH5TNCRRD76C6VEXEFR47FQA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.991339614 (625h31m52.862422659s) [noise=true , address=145.79.227.24:4583 , piece_id=WE5VKQ2HATUM6NRZJT3PSRHFZP74UFRS23SJA6LLL63VBCQRRMSQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.993748692 (625h31m52.864831747s) [noise=true , address=145.79.227.10:4401 , piece_id=6LT37OSUXSWCDPUEZUUYD2BPOTRQVPLJZSJPYXMAXTHXMJQ6CLWQ, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.993776514 (625h31m52.864859559s) [noise=true , address=145.79.227.13:4804 , piece_id=2RBUZRQIB2HW3OMTZO2BDPMCKRADGAODX2GOPR2LAC7WFJ3SZSTA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:52.995322709 (625h31m52.866405754s) [noise=true , address=145.79.227.6:4259 , piece_id=4YAYCJAFK6YO6BY6LS3XEKN3NWMCCGX7RMZKFLFPSGDGF5SVTPKA, seg_pos={0 0}, upload=548130]: Uploading piece... Dec 31 17:00:54.586869504 (625h31m54.457952549s) [upload=548130]: Next returned nil segment Dec 31 17:00:54.586887639 (625h31m54.457970684s) [upload=548130]: Waiting for error group managing segments... Dec 31 17:00:54.969318389 (625h31m54.840401434s) [noise=true , address=145.79.227.37:4565 , piece_id=O3KYPTHBUNLXDZDQJODLDXH7A5IVQ7G6OWFMUXWAX7YBUSF5MANQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.970514227 (625h31m54.841597272s) [noise=true , address=145.79.227.32:4835 , piece_id=ALG2N26FSRVDPQN6YOU5DKORV2ZPQRQBVK5XD3FQNRG7XLE2JBUA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.976748708 (625h31m54.847831754s) [noise=true , address=145.79.227.7:4295 , piece_id=SDH27VCHICBL5LULARSOOAHUQ5EVENKGDSPIY3IZYUDHZOIMSQXQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.979387142 (625h31m54.850470177s) [noise=true , address=145.79.227.15:4875 , piece_id=DYIJKALM4UAH5GHGX4QD6B3WLTA6F2N3MQLMI5AOEZ7D33ZPNDRA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.994174269 (625h31m54.865257314s) [noise=true , address=145.79.227.13:4804 , piece_id=2RBUZRQIB2HW3OMTZO2BDPMCKRADGAODX2GOPR2LAC7WFJ3SZSTA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:54.996871525 (625h31m54.86795457s) [noise=true , address=145.79.227.46:4602 , piece_id=4W5SBL3XEV7JYBGP5YZ62TWRDOBYFLJ3ENL3DX44IU4KYSSBQN7A, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.003898666 (625h31m54.874981731s) [noise=true , address=145.79.227.27:4691 , piece_id=5ICQCX373LFSVDQHKES6QQKIWGZ2LNSH25Y5FEC23MUYFYQNTXUA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.005164556 (625h31m54.876247611s) [noise=true , address=145.79.227.6:4259 , piece_id=4YAYCJAFK6YO6BY6LS3XEKN3NWMCCGX7RMZKFLFPSGDGF5SVTPKA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.005408150 (625h31m54.876491215s) [noise=true , address=145.79.227.24:4583 , piece_id=WE5VKQ2HATUM6NRZJT3PSRHFZP74UFRS23SJA6LLL63VBCQRRMSQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.010608472 (625h31m54.881691507s) [noise=true , address=145.79.227.10:4401 , piece_id=6LT37OSUXSWCDPUEZUUYD2BPOTRQVPLJZSJPYXMAXTHXMJQ6CLWQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.168517623 (625h31m55.039600668s) [noise=true , address=77.90.31.14:4661 , piece_id=U6F656GKPNRHZIMAU7EZWP2LGDAPQOW26GQPGH75EKDFOSU6HZMQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.319279103 (625h31m55.190362148s) [noise=true , address=45.91.116.12:4227 , piece_id=C4HDHTWJEK6HORMQ6WIK7TR4MVJ4GHHHCP56K2IDJG7O4F23GXEQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.518274919 (625h31m55.389357964s) [noise=true , address=45.91.116.9:4261 , piece_id=67O37BQPRO5VB5Z5BXNRTAHGE7IW4G7NNWFDBRMTZU3I6FS4IWJA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: <nil> Dec 31 17:00:55.518335765 (625h31m55.3894188s) [seg_pos={0 0}, upload=548130]: Segment reached optimal threshold of 13 pieces. Dec 31 17:00:55.518502642 (625h31m55.389585687s) [noise=true , address=145.79.227.40:4051 , piece_id=QRZONR6L4NW7LVT7BLTXLOQVRZ7KQI67YO5INA4YBULIKZAAXTPA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 17:00:55.518508594 (625h31m55.389591639s) [noise=true , address=145.79.227.40:4051 , piece_id=QRZONR6L4NW7LVT7BLTXLOQVRZ7KQI67YO5INA4YBULIKZAAXTPA, seg_pos={0 0}, upload=548130]: Long tail cancellation due to optimal threshold Dec 31 17:00:55.518554912 (625h31m55.389637947s) [noise=true , address=77.90.31.18:4805 , piece_id=275ZPWN4XS5NE7WBVB7ZYNJRGD6KR7LXKWNTNB7BK3PFAEMSI6HQ, seg_pos={0 0}, upload=548130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 31 17:00:55.518565051 (625h31m55.389648086s) [noise=true , address=77.90.31.18:4805 , piece_id=275ZPWN4XS5NE7WBVB7ZYNJRGD6KR7LXKWNTNB7BK3PFAEMSI6HQ, seg_pos={0 0}, upload=548130]: Long tail cancellation due to optimal threshold Dec 31 17:00:55.518715357 (625h31m55.389798402s) [noise=true , address=77.90.31.21:4877 , piece_id=JPW5YOZODCYKVQSXWGE6ZQHAEN2HZH5TNCRRD76C6VEXEFR47FQA, seg_pos={0 0}, upload=548130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46340->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:46340->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46340->77.90.31.21:4877: use of closed network connection Dec 31 17:00:55.518722210 (625h31m55.389805245s) [noise=true , address=77.90.31.21:4877 , piece_id=JPW5YOZODCYKVQSXWGE6ZQHAEN2HZH5TNCRRD76C6VEXEFR47FQA, seg_pos={0 0}, upload=548130]: Long tail cancellation due to optimal threshold Dec 31 17:00:55.518746115 (625h31m55.38982915s) [seg_pos={0 0}, upload=548130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 17:00:55.518834514 (625h31m55.389917549s) [upload=548130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 17:00:55.518839984 (625h31m55.389923029s) [upload=548130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 17:00:55.813662567 (625h31m55.684745622s) [upload=548130]: Done uploading segments... Dec 31 17:00:55.813687064 (625h31m55.684770119s) [upload=548130]: Upload finished. err: <nil> Duration: 30.003118114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.....................................X......X...X] duration: 1m0.006464493s trace: https://jaeger.dev.storj.io/trace/70b0cb39dd42d67b FAIL (2025-12-31 17:01:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.156432368 (625h31m18.027515413s) [noise=true , address=77.90.31.15:4710 , piece_id=AOXLJRW7BEWM7I3VFR64ONRKI3SX5SSUDH47SAXA3IAUPSRPZAOQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157072997 (625h31m18.028156042s) [noise=true , address=145.79.227.52:4459 , piece_id=OIZHPVYAWOUFB6EGGW65AYJA7CD2OJHPP5BV7X2BKON6OFSULSRA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157213074 (625h31m18.028296159s) [noise=true , address=77.90.31.15:4723 , piece_id=CKPWFCY5N3ZCZUBILOS3RZOS6AEFLWN4GH65L25TVYZAXZGQMWVA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157552671 (625h31m18.028635716s) [noise=true , address=77.90.31.17:4786 , piece_id=C6HSG4XIOQDNTB2RJTJOMFYOMDFDRF4MG2XFAPRSPS726NNBUTLQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.157787158 (625h31m18.028870203s) [noise=true , address=145.79.227.40:4072 , piece_id=UE56KE2MGKKROEX4M75BSK4WXD2IH2TDL3DVVFVDPTPX2HRQI4OQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158098741 (625h31m18.029181786s) [noise=true , address=77.90.31.13:4625 , piece_id=DPFGEZR5J2WRRNTISDYXJKWUTZSTRYDY2BS25HTOXQHVRZL52TRA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158386689 (625h31m18.029469734s) [noise=true , address=145.79.227.49:4343 , piece_id=UFWFBVKIWXCN6QMKXV3D3DCLPEV3DXW27N3J5AK3JWSJ2LXQZAIA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.158597901 (625h31m18.029680946s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.159008944 (625h31m18.030091989s) [noise=true , address=145.79.227.30:4779 , piece_id=HIJQGU3DHFJHVR3ISVSWPZBDX5Y2Q5N4LTUM43GOXCOURUV2AJTQ, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.159761006 (625h31m18.030844051s) [noise=true , address=145.79.227.49:4345 , piece_id=RIPCK32N6WLUQYS7EPK6TSEEAQH2DUVO3JZ4VVW3SJSCRCRWZE2A, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.160164584 (625h31m18.03124763s) [noise=true , address=145.79.227.34:4937 , piece_id=II2LUE2FIDSVNDIBQEF5G2ISPFGL5CFC6NLENMICICJC7HHSPIOA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.160727346 (625h31m18.031810391s) [noise=true , address=145.79.227.33:4889 , piece_id=S34ENDYWNCHS2TLSPVG3WTIBQ2QZ5DV2O3VX76C3IWWEUGZDF2YA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:18.161214243 (625h31m18.032297288s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Uploading piece... Dec 31 17:00:19.631860285 (625h31m19.50294335s) [upload=548124]: Next returned nil segment Dec 31 17:00:19.631881466 (625h31m19.502964501s) [upload=548124]: Waiting for error group managing segments... Dec 31 17:00:20.077955152 (625h31m19.949038197s) [noise=true , address=145.79.227.52:4459 , piece_id=OIZHPVYAWOUFB6EGGW65AYJA7CD2OJHPP5BV7X2BKON6OFSULSRA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.196234912 (625h31m20.067317957s) [noise=true , address=77.90.31.17:4786 , piece_id=C6HSG4XIOQDNTB2RJTJOMFYOMDFDRF4MG2XFAPRSPS726NNBUTLQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.198807942 (625h31m20.069890987s) [noise=true , address=77.90.31.15:4710 , piece_id=AOXLJRW7BEWM7I3VFR64ONRKI3SX5SSUDH47SAXA3IAUPSRPZAOQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.213081430 (625h31m20.084164485s) [noise=true , address=77.90.31.12:4599 , piece_id=QZR3U7P2RGBTEB2LNIOJB6XSRM2NW2KNWPQAB6UR3H6MRRMK7WUQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.285928892 (625h31m20.157011938s) [noise=true , address=145.79.227.40:4072 , piece_id=UE56KE2MGKKROEX4M75BSK4WXD2IH2TDL3DVVFVDPTPX2HRQI4OQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.366975977 (625h31m20.238059022s) [noise=true , address=45.91.116.9:4288 , piece_id=BFB6Z3MDIKZ7AKRLPT6PWQESMDMXHYAWDMN5YCUJXML7WZVAMWOQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.399993577 (625h31m20.271076612s) [noise=true , address=77.90.31.13:4625 , piece_id=DPFGEZR5J2WRRNTISDYXJKWUTZSTRYDY2BS25HTOXQHVRZL52TRA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.977929104 (625h31m20.84901214s) [noise=true , address=145.79.227.44:4204 , piece_id=E55ISFZIJLDZKCIVB7OCVNJZCUJFCWRDKSRDTEZWBLF4FPDNW5BA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:20.984855463 (625h31m20.855938499s) [noise=true , address=77.90.31.15:4723 , piece_id=CKPWFCY5N3ZCZUBILOS3RZOS6AEFLWN4GH65L25TVYZAXZGQMWVA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.012859246 (625h31m20.883942291s) [noise=true , address=145.79.227.49:4345 , piece_id=RIPCK32N6WLUQYS7EPK6TSEEAQH2DUVO3JZ4VVW3SJSCRCRWZE2A, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.073282851 (625h31m20.944365896s) [noise=true , address=145.79.227.30:4779 , piece_id=HIJQGU3DHFJHVR3ISVSWPZBDX5Y2Q5N4LTUM43GOXCOURUV2AJTQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.132514036 (625h31m21.003597081s) [noise=true , address=145.79.227.49:4343 , piece_id=UFWFBVKIWXCN6QMKXV3D3DCLPEV3DXW27N3J5AK3JWSJ2LXQZAIA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.154431609 (625h31m21.025514664s) [noise=true , address=145.79.227.33:4889 , piece_id=S34ENDYWNCHS2TLSPVG3WTIBQ2QZ5DV2O3VX76C3IWWEUGZDF2YA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.456343772 (625h31m21.327426817s) [noise=true , address=145.79.227.34:4937 , piece_id=II2LUE2FIDSVNDIBQEF5G2ISPFGL5CFC6NLENMICICJC7HHSPIOA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: <nil> Dec 31 17:00:21.456381544 (625h31m21.327464589s) [seg_pos={0 0}, upload=548124]: Segment reached optimal threshold of 14 pieces. Dec 31 17:00:21.456811993 (625h31m21.327895059s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection; write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40354->145.79.227.27:4708: use of closed network connection Dec 31 17:00:21.456824007 (625h31m21.327907052s) [noise=true , address=145.79.227.27:4708 , piece_id=7INDDQAUD6VZDWNZD4JPP6OR22XD7X3FV24AVH56JR62GILL6FJA, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457303460 (625h31m21.328386505s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1onAnFbXC1LbUAD31oHGp75v9DpnWmx8LTe6Wazf7XuR3rKnJw): protocol: write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection; write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37114->145.79.227.48:4325: use of closed network connection Dec 31 17:00:21.457313459 (625h31m21.328396494s) [noise=true , address=145.79.227.48:4325 , piece_id=G7MFCEMBEQNMZZAFBKEOKKYSCFA6QNW2BJGX7UNKSK7OGMKTV2MQ, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457441663 (625h31m21.328524708s) [noise=true , address=145.79.227.6:4266 , piece_id=GE4XNR6X3DYDYZFUAYZ3X3YVCZ3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wH8jka9Jw8JLYdGKuit8XoJjVLJRDKJ5ePgeei16eax6mk3j): protocol: write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection; write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33884->145.79.227.6:4266: use of closed network connection Dec 31 17:00:21.457512669 (625h31m21.328595714s) [noise=true , address=145.79.227.6:4266 , piece_id=GE4XNR6X3DYDYZFUAYZ3X3YVCZ3J6PGCUGH7ZGCJERXWMR6EJCHQ, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457641062 (625h31m21.328724107s) [noise=true , address=145.79.227.51:4441 , piece_id=3SEJYHIUPVRHP5QVOWZRB3SBFYVZ7OMRHH3HXQCFGWEZMMCPG5BA, seg_pos={0 0}, upload=548124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VBayWisMK1n3cwypeiLzGuffonBTzYga7thAZ8pSqrusc7kow): protocol: expected piece hash; context canceled; EOF Dec 31 17:00:21.457649288 (625h31m21.328732333s) [noise=true , address=145.79.227.51:4441 , piece_id=3SEJYHIUPVRHP5QVOWZRB3SBFYVZ7OMRHH3HXQCFGWEZMMCPG5BA, seg_pos={0 0}, upload=548124]: Long tail cancellation due to optimal threshold Dec 31 17:00:21.457667453 (625h31m21.328765977s) [seg_pos={0 0}, upload=548124]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 17:00:21.457773294 (625h31m21.328856339s) [upload=548124]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 17:00:21.457780127 (625h31m21.328863172s) [upload=548124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 17:00:21.754059214 (625h31m21.625142259s) [upload=548124]: Done uploading segments... Dec 31 17:00:21.754081627 (625h31m21.625164662s) [upload=548124]: Upload finished. err: <nil> Duration: 1m0.006487648s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.XXX...X] duration: 5.000944099s trace: https://jaeger.dev.storj.io/trace/71adf0745edb131f FAIL (2025-12-31 17:01:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000970129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.XXX...] duration: 3.629765684s trace: https://jaeger.dev.storj.io/trace/62ef902d4708d459 FAIL (2025-12-31 17:00:29 UTC) orbiter warning: context deadline exceeded Duration: 3.629802102s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.XXX..] duration: 4.484505225s trace: https://jaeger.dev.storj.io/trace/62ef902d4708d459 FAIL (2025-12-31 16:59:58 UTC) orbiter warning: context deadline exceeded Duration: 4.484535895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X....X..] duration: 7.088484758s trace: https://jaeger.dev.storj.io/trace/2670c1d470b783c0 FAIL (2025-12-31 16:59:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ) context deadline exceeded ======LOGS====== Dec 31 16:57:47.565786811 (625h28m47.436869856s) [upload=548085]: Starting upload Dec 31 16:57:47.565890649 (625h28m47.436973694s) [upload=548085]: Done starting upload Dec 31 16:57:47.566112552 (625h28m47.437195597s) [upload=548085]: Uploading segments... Dec 31 16:57:47.566156626 (625h28m47.437239681s) [upload=548085]: Got next segment. Inline: false Dec 31 16:57:47.566185250 (625h28m47.437268296s) [upload=548085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 16:57:47.566190200 (625h28m47.437273235s) [upload=548085]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 16:57:47.812075209 (625h28m47.683158244s) [seg_pos={0 0}, upload=548085]: Begin upload segment... Dec 31 16:57:47.812130525 (625h28m47.68321356s) [seg_pos={0 0}, upload=548085]: Done begin upload segment. Dec 31 16:57:47.812164800 (625h28m47.683247835s) [noise=true , address=145.79.227.13:4804 , piece_id=AAUFXV73JNGZBTMKBA3UNMVTVZMD6A3A6N5K7AU4TVSCMD3PXKUQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.812222480 (625h28m47.683305515s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813060045 (625h28m47.68414308s) [noise=true , address=145.79.227.27:4689 , piece_id=A44ST6SCYFVQENJFAVL6WDFOHLI5R2HMLYTJAI2B5XAECCF7YQJQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813444006 (625h28m47.684527041s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813676348 (625h28m47.684759383s) [noise=true , address=145.79.227.5:4221 , piece_id=5XVHU4GOLP4BOEEOQ7YNLQ37LNCCBLIWIOU6BFVBROUH6X66MM7A, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813843206 (625h28m47.684926241s) [noise=true , address=145.79.227.2:4113 , piece_id=GV72GLTUUXXGVYDMLRKV2QBXOE7IRN5MKCWJMN4KPCHIDJUV73NQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814080999 (625h28m47.685164034s) [noise=true , address=77.90.31.10:4519 , piece_id=F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814381431 (625h28m47.685464476s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814416498 (625h28m47.685499533s) [noise=true , address=145.79.227.18:4006 , piece_id=PCRVVFQ2WYWLWKSYCVV462GFGJLKWTXDOZM5N6NII6LT3F34YTRQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814870462 (625h28m47.685953507s) [noise=true , address=145.79.227.22:4511 , piece_id=FIT2W6CM5Q77PSAWSPLOXDM2XD56NLDK5HLDEWY2A6NN3O4AQQAQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814878157 (625h28m47.685961202s) [noise=true , address=145.79.227.9:4367 , piece_id=AOSHODY5MHZSG6EMAKEUDPKNDPZVULERG6NXVQWLSWHSC5UGO3IA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.815238572 (625h28m47.686321608s) [noise=true , address=77.90.31.19:4841 , piece_id=XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.816120533 (625h28m47.687203578s) [noise=true , address=145.79.227.37:4567 , piece_id=EKKPOHUXXH437ZQ67K2LOPZ5773NJXLRIQTAKM7E7BTMRV5LLTJA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.816236333 (625h28m47.687319368s) [noise=true , address=145.79.227.25:4619 , piece_id=MWXQL4ZDVJ2DO53MJQPPNO7EUFCQN6JYB5YSHQMKDBJRVEBRA3AA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.817276784 (625h28m47.688359819s) [noise=true , address=45.91.116.8:4048 , piece_id=6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.819322338 (625h28m47.690405373s) [noise=true , address=145.79.227.4:4187 , piece_id=53VLJWVLEWEFAVUKJVOGS6WCHUCA2APCUGRGA5VH4PDELLNU5EGA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:49.512295273 (625h28m49.383378308s) [upload=548085]: Next returned nil segment Dec 31 16:57:49.512308427 (625h28m49.383391462s) [upload=548085]: Waiting for error group managing segments... Dec 31 16:57:49.787063029 (625h28m49.658146074s) [noise=true , address=145.79.227.13:4804 , piece_id=AAUFXV73JNGZBTMKBA3UNMVTVZMD6A3A6N5K7AU4TVSCMD3PXKUQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.794380723 (625h28m49.665463768s) [noise=true , address=145.79.227.2:4113 , piece_id=GV72GLTUUXXGVYDMLRKV2QBXOE7IRN5MKCWJMN4KPCHIDJUV73NQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.801155382 (625h28m49.672238417s) [noise=true , address=145.79.227.27:4689 , piece_id=A44ST6SCYFVQENJFAVL6WDFOHLI5R2HMLYTJAI2B5XAECCF7YQJQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.805611888 (625h28m49.676694923s) [noise=true , address=145.79.227.18:4006 , piece_id=PCRVVFQ2WYWLWKSYCVV462GFGJLKWTXDOZM5N6NII6LT3F34YTRQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.818638573 (625h28m49.689721608s) [noise=true , address=145.79.227.25:4619 , piece_id=MWXQL4ZDVJ2DO53MJQPPNO7EUFCQN6JYB5YSHQMKDBJRVEBRA3AA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.823609939 (625h28m49.694693005s) [noise=true , address=145.79.227.22:4511 , piece_id=FIT2W6CM5Q77PSAWSPLOXDM2XD56NLDK5HLDEWY2A6NN3O4AQQAQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.825557458 (625h28m49.696640493s) [noise=true , address=145.79.227.4:4187 , piece_id=53VLJWVLEWEFAVUKJVOGS6WCHUCA2APCUGRGA5VH4PDELLNU5EGA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.829627417 (625h28m49.700710452s) [noise=true , address=145.79.227.37:4567 , piece_id=EKKPOHUXXH437ZQ67K2LOPZ5773NJXLRIQTAKM7E7BTMRV5LLTJA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.829826126 (625h28m49.700909171s) [noise=true , address=145.79.227.9:4367 , piece_id=AOSHODY5MHZSG6EMAKEUDPKNDPZVULERG6NXVQWLSWHSC5UGO3IA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.972918433 (625h28m49.844001468s) [noise=true , address=145.79.227.5:4221 , piece_id=5XVHU4GOLP4BOEEOQ7YNLQ37LNCCBLIWIOU6BFVBROUH6X66MM7A, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.016337258 (625h28m49.887420303s) [noise=true , address=77.90.31.19:4841 , piece_id=XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.066606578 (625h28m49.937689614s) [noise=true , address=77.90.31.10:4519 , piece_id=F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.802894050 (625h28m50.673977085s) [noise=true , address=45.91.116.8:4048 , piece_id=6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.802917805 (625h28m50.67400085s) [seg_pos={0 0}, upload=548085]: Segment reached optimal threshold of 13 pieces. Dec 31 16:57:50.803136832 (625h28m50.674219877s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 31 16:57:50.803144076 (625h28m50.674227121s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold Dec 31 16:57:50.803253285 (625h28m50.67433633s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 31 16:57:50.803258725 (625h28m50.67434177s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold Dec 31 16:57:50.803484023 (625h28m50.674567068s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 16:57:50.803502799 (625h28m50.674585845s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold Dec 31 16:57:50.803532987 (625h28m50.674616032s) [seg_pos={0 0}, upload=548085]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 16:57:50.803658395 (625h28m50.67474145s) [upload=548085]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:57:50.803666380 (625h28m50.674749415s) [upload=548085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:57:51.091543468 (625h28m50.962626503s) [upload=548085]: Done uploading segments... Dec 31 16:57:51.091565551 (625h28m50.962648586s) [upload=548085]: Upload finished. err: <nil> Duration: 7.088513713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..........X.....................................X......X...] duration: 38.856760872s trace: https://jaeger.dev.storj.io/trace/51209bc25994253c FAIL (2025-12-31 16:59:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.882880440 (625h27m34.753963475s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.883312763 (625h27m34.754395808s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.883748182 (625h27m34.754831227s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884108909 (625h27m34.755191944s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884134819 (625h27m34.755217854s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884545450 (625h27m34.755628495s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884570638 (625h27m34.755653683s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884987812 (625h27m34.756070847s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885002600 (625h27m34.756085635s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885364570 (625h27m34.756447615s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885437358 (625h27m34.756520394s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885713234 (625h27m34.756796269s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886124126 (625h27m34.757207172s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886498589 (625h27m34.757581624s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886844768 (625h27m34.757927803s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:36.347113954 (625h27m36.218196999s) [upload=548072]: Next returned nil segment Dec 31 16:56:36.347127068 (625h27m36.218210114s) [upload=548072]: Waiting for error group managing segments... Dec 31 16:56:36.782391943 (625h27m36.653474978s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.828828685 (625h27m36.69991172s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.909133386 (625h27m36.780216421s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.951641147 (625h27m36.822724192s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.953178845 (625h27m36.82426188s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.967346992 (625h27m36.838430027s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.015751392 (625h27m36.886834427s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.070458612 (625h27m36.941541647s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.277203082 (625h27m37.148286118s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.620497854 (625h27m37.491580909s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.678713946 (625h27m37.549796991s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.686079791 (625h27m37.557162836s) [noise=true , address=145.79.227.33:4878 , piece_id=N5PTWWSHXTUK4LBAWQ5272RLIBLSCSFKML2DSCIVQCQN7NO7KFPQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.719337278 (625h27m37.590420313s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.738954522 (625h27m37.610037557s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.738974741 (625h27m37.610057776s) [seg_pos={0 0}, upload=548072]: Segment reached optimal threshold of 14 pieces. Dec 31 16:56:37.739210258 (625h27m37.610293293s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rm4HueKsNdwL3eCTCSWoPseR2LQWcsE3jNPpxSfYQQorj9LL): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739218525 (625h27m37.61030157s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739273609 (625h27m37.610356654s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739292536 (625h27m37.610375581s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739499520 (625h27m37.610582565s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739506242 (625h27m37.610589277s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739518896 (625h27m37.610601931s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection Dec 31 16:56:37.739527362 (625h27m37.610610407s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739545647 (625h27m37.610628692s) [seg_pos={0 0}, upload=548072]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 16:56:37.739638374 (625h27m37.610721409s) [upload=548072]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:56:37.739643183 (625h27m37.610726228s) [upload=548072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:56:38.030777232 (625h27m37.901860267s) [upload=548072]: Done uploading segments... Dec 31 16:56:38.030797912 (625h27m37.901880947s) [upload=548072]: Upload finished. err: <nil> Duration: 38.856794796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.XXX.] duration: 3.594123698s trace: https://jaeger.dev.storj.io/trace/62ef902d4708d459 FAIL (2025-12-31 16:59:08 UTC) orbiter warning: context deadline exceeded Duration: 3.594151571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..........X.....................................X......X..] duration: 41.454237029s trace: https://jaeger.dev.storj.io/trace/51209bc25994253c FAIL (2025-12-31 16:59:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.882880440 (625h27m34.753963475s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.883312763 (625h27m34.754395808s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.883748182 (625h27m34.754831227s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884108909 (625h27m34.755191944s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884134819 (625h27m34.755217854s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884545450 (625h27m34.755628495s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884570638 (625h27m34.755653683s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884987812 (625h27m34.756070847s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885002600 (625h27m34.756085635s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885364570 (625h27m34.756447615s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885437358 (625h27m34.756520394s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885713234 (625h27m34.756796269s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886124126 (625h27m34.757207172s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886498589 (625h27m34.757581624s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886844768 (625h27m34.757927803s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:36.347113954 (625h27m36.218196999s) [upload=548072]: Next returned nil segment Dec 31 16:56:36.347127068 (625h27m36.218210114s) [upload=548072]: Waiting for error group managing segments... Dec 31 16:56:36.782391943 (625h27m36.653474978s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.828828685 (625h27m36.69991172s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.909133386 (625h27m36.780216421s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.951641147 (625h27m36.822724192s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.953178845 (625h27m36.82426188s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.967346992 (625h27m36.838430027s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.015751392 (625h27m36.886834427s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.070458612 (625h27m36.941541647s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.277203082 (625h27m37.148286118s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.620497854 (625h27m37.491580909s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.678713946 (625h27m37.549796991s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.686079791 (625h27m37.557162836s) [noise=true , address=145.79.227.33:4878 , piece_id=N5PTWWSHXTUK4LBAWQ5272RLIBLSCSFKML2DSCIVQCQN7NO7KFPQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.719337278 (625h27m37.590420313s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.738954522 (625h27m37.610037557s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.738974741 (625h27m37.610057776s) [seg_pos={0 0}, upload=548072]: Segment reached optimal threshold of 14 pieces. Dec 31 16:56:37.739210258 (625h27m37.610293293s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rm4HueKsNdwL3eCTCSWoPseR2LQWcsE3jNPpxSfYQQorj9LL): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739218525 (625h27m37.61030157s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739273609 (625h27m37.610356654s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739292536 (625h27m37.610375581s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739499520 (625h27m37.610582565s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739506242 (625h27m37.610589277s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739518896 (625h27m37.610601931s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection Dec 31 16:56:37.739527362 (625h27m37.610610407s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739545647 (625h27m37.610628692s) [seg_pos={0 0}, upload=548072]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 16:56:37.739638374 (625h27m37.610721409s) [upload=548072]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:56:37.739643183 (625h27m37.610726228s) [upload=548072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:56:38.030777232 (625h27m37.901860267s) [upload=548072]: Done uploading segments... Dec 31 16:56:38.030797912 (625h27m37.901880947s) [upload=548072]: Upload finished. err: <nil> Duration: 41.454253822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.......X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X....X.] duration: 6.054595921s trace: https://jaeger.dev.storj.io/trace/2670c1d470b783c0 FAIL (2025-12-31 16:58:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ) context deadline exceeded ======LOGS====== Dec 31 16:57:47.565786811 (625h28m47.436869856s) [upload=548085]: Starting upload Dec 31 16:57:47.565890649 (625h28m47.436973694s) [upload=548085]: Done starting upload Dec 31 16:57:47.566112552 (625h28m47.437195597s) [upload=548085]: Uploading segments... Dec 31 16:57:47.566156626 (625h28m47.437239681s) [upload=548085]: Got next segment. Inline: false Dec 31 16:57:47.566185250 (625h28m47.437268296s) [upload=548085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 16:57:47.566190200 (625h28m47.437273235s) [upload=548085]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 16:57:47.812075209 (625h28m47.683158244s) [seg_pos={0 0}, upload=548085]: Begin upload segment... Dec 31 16:57:47.812130525 (625h28m47.68321356s) [seg_pos={0 0}, upload=548085]: Done begin upload segment. Dec 31 16:57:47.812164800 (625h28m47.683247835s) [noise=true , address=145.79.227.13:4804 , piece_id=AAUFXV73JNGZBTMKBA3UNMVTVZMD6A3A6N5K7AU4TVSCMD3PXKUQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.812222480 (625h28m47.683305515s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813060045 (625h28m47.68414308s) [noise=true , address=145.79.227.27:4689 , piece_id=A44ST6SCYFVQENJFAVL6WDFOHLI5R2HMLYTJAI2B5XAECCF7YQJQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813444006 (625h28m47.684527041s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813676348 (625h28m47.684759383s) [noise=true , address=145.79.227.5:4221 , piece_id=5XVHU4GOLP4BOEEOQ7YNLQ37LNCCBLIWIOU6BFVBROUH6X66MM7A, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813843206 (625h28m47.684926241s) [noise=true , address=145.79.227.2:4113 , piece_id=GV72GLTUUXXGVYDMLRKV2QBXOE7IRN5MKCWJMN4KPCHIDJUV73NQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814080999 (625h28m47.685164034s) [noise=true , address=77.90.31.10:4519 , piece_id=F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814381431 (625h28m47.685464476s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814416498 (625h28m47.685499533s) [noise=true , address=145.79.227.18:4006 , piece_id=PCRVVFQ2WYWLWKSYCVV462GFGJLKWTXDOZM5N6NII6LT3F34YTRQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814870462 (625h28m47.685953507s) [noise=true , address=145.79.227.22:4511 , piece_id=FIT2W6CM5Q77PSAWSPLOXDM2XD56NLDK5HLDEWY2A6NN3O4AQQAQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814878157 (625h28m47.685961202s) [noise=true , address=145.79.227.9:4367 , piece_id=AOSHODY5MHZSG6EMAKEUDPKNDPZVULERG6NXVQWLSWHSC5UGO3IA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.815238572 (625h28m47.686321608s) [noise=true , address=77.90.31.19:4841 , piece_id=XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.816120533 (625h28m47.687203578s) [noise=true , address=145.79.227.37:4567 , piece_id=EKKPOHUXXH437ZQ67K2LOPZ5773NJXLRIQTAKM7E7BTMRV5LLTJA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.816236333 (625h28m47.687319368s) [noise=true , address=145.79.227.25:4619 , piece_id=MWXQL4ZDVJ2DO53MJQPPNO7EUFCQN6JYB5YSHQMKDBJRVEBRA3AA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.817276784 (625h28m47.688359819s) [noise=true , address=45.91.116.8:4048 , piece_id=6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.819322338 (625h28m47.690405373s) [noise=true , address=145.79.227.4:4187 , piece_id=53VLJWVLEWEFAVUKJVOGS6WCHUCA2APCUGRGA5VH4PDELLNU5EGA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:49.512295273 (625h28m49.383378308s) [upload=548085]: Next returned nil segment Dec 31 16:57:49.512308427 (625h28m49.383391462s) [upload=548085]: Waiting for error group managing segments... Dec 31 16:57:49.787063029 (625h28m49.658146074s) [noise=true , address=145.79.227.13:4804 , piece_id=AAUFXV73JNGZBTMKBA3UNMVTVZMD6A3A6N5K7AU4TVSCMD3PXKUQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.794380723 (625h28m49.665463768s) [noise=true , address=145.79.227.2:4113 , piece_id=GV72GLTUUXXGVYDMLRKV2QBXOE7IRN5MKCWJMN4KPCHIDJUV73NQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.801155382 (625h28m49.672238417s) [noise=true , address=145.79.227.27:4689 , piece_id=A44ST6SCYFVQENJFAVL6WDFOHLI5R2HMLYTJAI2B5XAECCF7YQJQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.805611888 (625h28m49.676694923s) [noise=true , address=145.79.227.18:4006 , piece_id=PCRVVFQ2WYWLWKSYCVV462GFGJLKWTXDOZM5N6NII6LT3F34YTRQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.818638573 (625h28m49.689721608s) [noise=true , address=145.79.227.25:4619 , piece_id=MWXQL4ZDVJ2DO53MJQPPNO7EUFCQN6JYB5YSHQMKDBJRVEBRA3AA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.823609939 (625h28m49.694693005s) [noise=true , address=145.79.227.22:4511 , piece_id=FIT2W6CM5Q77PSAWSPLOXDM2XD56NLDK5HLDEWY2A6NN3O4AQQAQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.825557458 (625h28m49.696640493s) [noise=true , address=145.79.227.4:4187 , piece_id=53VLJWVLEWEFAVUKJVOGS6WCHUCA2APCUGRGA5VH4PDELLNU5EGA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.829627417 (625h28m49.700710452s) [noise=true , address=145.79.227.37:4567 , piece_id=EKKPOHUXXH437ZQ67K2LOPZ5773NJXLRIQTAKM7E7BTMRV5LLTJA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.829826126 (625h28m49.700909171s) [noise=true , address=145.79.227.9:4367 , piece_id=AOSHODY5MHZSG6EMAKEUDPKNDPZVULERG6NXVQWLSWHSC5UGO3IA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.972918433 (625h28m49.844001468s) [noise=true , address=145.79.227.5:4221 , piece_id=5XVHU4GOLP4BOEEOQ7YNLQ37LNCCBLIWIOU6BFVBROUH6X66MM7A, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.016337258 (625h28m49.887420303s) [noise=true , address=77.90.31.19:4841 , piece_id=XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.066606578 (625h28m49.937689614s) [noise=true , address=77.90.31.10:4519 , piece_id=F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.802894050 (625h28m50.673977085s) [noise=true , address=45.91.116.8:4048 , piece_id=6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.802917805 (625h28m50.67400085s) [seg_pos={0 0}, upload=548085]: Segment reached optimal threshold of 13 pieces. Dec 31 16:57:50.803136832 (625h28m50.674219877s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 31 16:57:50.803144076 (625h28m50.674227121s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold Dec 31 16:57:50.803253285 (625h28m50.67433633s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 31 16:57:50.803258725 (625h28m50.67434177s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold Dec 31 16:57:50.803484023 (625h28m50.674567068s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 16:57:50.803502799 (625h28m50.674585845s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold Dec 31 16:57:50.803532987 (625h28m50.674616032s) [seg_pos={0 0}, upload=548085]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 16:57:50.803658395 (625h28m50.67474145s) [upload=548085]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:57:50.803666380 (625h28m50.674749415s) [upload=548085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:57:51.091543468 (625h28m50.962626503s) [upload=548085]: Done uploading segments... Dec 31 16:57:51.091565551 (625h28m50.962648586s) [upload=548085]: Upload finished. err: <nil> Duration: 6.054621941s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.XXX] duration: 5.001685104s trace: https://jaeger.dev.storj.io/trace/62ef902d4708d459 FAIL (2025-12-31 16:58:49 UTC) orbiter warning: context deadline exceeded Duration: 5.001709392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X..........X.....................................X......X.] duration: 47.499427281s trace: https://jaeger.dev.storj.io/trace/51209bc25994253c FAIL (2025-12-31 16:58:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.882880440 (625h27m34.753963475s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.883312763 (625h27m34.754395808s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.883748182 (625h27m34.754831227s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884108909 (625h27m34.755191944s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884134819 (625h27m34.755217854s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884545450 (625h27m34.755628495s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884570638 (625h27m34.755653683s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884987812 (625h27m34.756070847s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885002600 (625h27m34.756085635s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885364570 (625h27m34.756447615s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885437358 (625h27m34.756520394s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885713234 (625h27m34.756796269s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886124126 (625h27m34.757207172s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886498589 (625h27m34.757581624s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886844768 (625h27m34.757927803s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:36.347113954 (625h27m36.218196999s) [upload=548072]: Next returned nil segment Dec 31 16:56:36.347127068 (625h27m36.218210114s) [upload=548072]: Waiting for error group managing segments... Dec 31 16:56:36.782391943 (625h27m36.653474978s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.828828685 (625h27m36.69991172s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.909133386 (625h27m36.780216421s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.951641147 (625h27m36.822724192s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.953178845 (625h27m36.82426188s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.967346992 (625h27m36.838430027s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.015751392 (625h27m36.886834427s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.070458612 (625h27m36.941541647s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.277203082 (625h27m37.148286118s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.620497854 (625h27m37.491580909s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.678713946 (625h27m37.549796991s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.686079791 (625h27m37.557162836s) [noise=true , address=145.79.227.33:4878 , piece_id=N5PTWWSHXTUK4LBAWQ5272RLIBLSCSFKML2DSCIVQCQN7NO7KFPQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.719337278 (625h27m37.590420313s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.738954522 (625h27m37.610037557s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.738974741 (625h27m37.610057776s) [seg_pos={0 0}, upload=548072]: Segment reached optimal threshold of 14 pieces. Dec 31 16:56:37.739210258 (625h27m37.610293293s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rm4HueKsNdwL3eCTCSWoPseR2LQWcsE3jNPpxSfYQQorj9LL): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739218525 (625h27m37.61030157s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739273609 (625h27m37.610356654s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739292536 (625h27m37.610375581s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739499520 (625h27m37.610582565s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739506242 (625h27m37.610589277s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739518896 (625h27m37.610601931s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection Dec 31 16:56:37.739527362 (625h27m37.610610407s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739545647 (625h27m37.610628692s) [seg_pos={0 0}, upload=548072]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 16:56:37.739638374 (625h27m37.610721409s) [upload=548072]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:56:37.739643183 (625h27m37.610726228s) [upload=548072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:56:38.030777232 (625h27m37.901860267s) [upload=548072]: Done uploading segments... Dec 31 16:56:38.030797912 (625h27m37.901880947s) [upload=548072]: Upload finished. err: <nil> Duration: 47.499442329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X....X] duration: 30.003628245s trace: https://jaeger.dev.storj.io/trace/2670c1d470b783c0 FAIL (2025-12-31 16:58:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ) context deadline exceeded ======LOGS====== Dec 31 16:57:47.565786811 (625h28m47.436869856s) [upload=548085]: Starting upload Dec 31 16:57:47.565890649 (625h28m47.436973694s) [upload=548085]: Done starting upload Dec 31 16:57:47.566112552 (625h28m47.437195597s) [upload=548085]: Uploading segments... Dec 31 16:57:47.566156626 (625h28m47.437239681s) [upload=548085]: Got next segment. Inline: false Dec 31 16:57:47.566185250 (625h28m47.437268296s) [upload=548085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 16:57:47.566190200 (625h28m47.437273235s) [upload=548085]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 16:57:47.812075209 (625h28m47.683158244s) [seg_pos={0 0}, upload=548085]: Begin upload segment... Dec 31 16:57:47.812130525 (625h28m47.68321356s) [seg_pos={0 0}, upload=548085]: Done begin upload segment. Dec 31 16:57:47.812164800 (625h28m47.683247835s) [noise=true , address=145.79.227.13:4804 , piece_id=AAUFXV73JNGZBTMKBA3UNMVTVZMD6A3A6N5K7AU4TVSCMD3PXKUQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.812222480 (625h28m47.683305515s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813060045 (625h28m47.68414308s) [noise=true , address=145.79.227.27:4689 , piece_id=A44ST6SCYFVQENJFAVL6WDFOHLI5R2HMLYTJAI2B5XAECCF7YQJQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813444006 (625h28m47.684527041s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813676348 (625h28m47.684759383s) [noise=true , address=145.79.227.5:4221 , piece_id=5XVHU4GOLP4BOEEOQ7YNLQ37LNCCBLIWIOU6BFVBROUH6X66MM7A, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.813843206 (625h28m47.684926241s) [noise=true , address=145.79.227.2:4113 , piece_id=GV72GLTUUXXGVYDMLRKV2QBXOE7IRN5MKCWJMN4KPCHIDJUV73NQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814080999 (625h28m47.685164034s) [noise=true , address=77.90.31.10:4519 , piece_id=F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814381431 (625h28m47.685464476s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814416498 (625h28m47.685499533s) [noise=true , address=145.79.227.18:4006 , piece_id=PCRVVFQ2WYWLWKSYCVV462GFGJLKWTXDOZM5N6NII6LT3F34YTRQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814870462 (625h28m47.685953507s) [noise=true , address=145.79.227.22:4511 , piece_id=FIT2W6CM5Q77PSAWSPLOXDM2XD56NLDK5HLDEWY2A6NN3O4AQQAQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.814878157 (625h28m47.685961202s) [noise=true , address=145.79.227.9:4367 , piece_id=AOSHODY5MHZSG6EMAKEUDPKNDPZVULERG6NXVQWLSWHSC5UGO3IA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.815238572 (625h28m47.686321608s) [noise=true , address=77.90.31.19:4841 , piece_id=XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.816120533 (625h28m47.687203578s) [noise=true , address=145.79.227.37:4567 , piece_id=EKKPOHUXXH437ZQ67K2LOPZ5773NJXLRIQTAKM7E7BTMRV5LLTJA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.816236333 (625h28m47.687319368s) [noise=true , address=145.79.227.25:4619 , piece_id=MWXQL4ZDVJ2DO53MJQPPNO7EUFCQN6JYB5YSHQMKDBJRVEBRA3AA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.817276784 (625h28m47.688359819s) [noise=true , address=45.91.116.8:4048 , piece_id=6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:47.819322338 (625h28m47.690405373s) [noise=true , address=145.79.227.4:4187 , piece_id=53VLJWVLEWEFAVUKJVOGS6WCHUCA2APCUGRGA5VH4PDELLNU5EGA, seg_pos={0 0}, upload=548085]: Uploading piece... Dec 31 16:57:49.512295273 (625h28m49.383378308s) [upload=548085]: Next returned nil segment Dec 31 16:57:49.512308427 (625h28m49.383391462s) [upload=548085]: Waiting for error group managing segments... Dec 31 16:57:49.787063029 (625h28m49.658146074s) [noise=true , address=145.79.227.13:4804 , piece_id=AAUFXV73JNGZBTMKBA3UNMVTVZMD6A3A6N5K7AU4TVSCMD3PXKUQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.794380723 (625h28m49.665463768s) [noise=true , address=145.79.227.2:4113 , piece_id=GV72GLTUUXXGVYDMLRKV2QBXOE7IRN5MKCWJMN4KPCHIDJUV73NQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.801155382 (625h28m49.672238417s) [noise=true , address=145.79.227.27:4689 , piece_id=A44ST6SCYFVQENJFAVL6WDFOHLI5R2HMLYTJAI2B5XAECCF7YQJQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.805611888 (625h28m49.676694923s) [noise=true , address=145.79.227.18:4006 , piece_id=PCRVVFQ2WYWLWKSYCVV462GFGJLKWTXDOZM5N6NII6LT3F34YTRQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.818638573 (625h28m49.689721608s) [noise=true , address=145.79.227.25:4619 , piece_id=MWXQL4ZDVJ2DO53MJQPPNO7EUFCQN6JYB5YSHQMKDBJRVEBRA3AA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.823609939 (625h28m49.694693005s) [noise=true , address=145.79.227.22:4511 , piece_id=FIT2W6CM5Q77PSAWSPLOXDM2XD56NLDK5HLDEWY2A6NN3O4AQQAQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.825557458 (625h28m49.696640493s) [noise=true , address=145.79.227.4:4187 , piece_id=53VLJWVLEWEFAVUKJVOGS6WCHUCA2APCUGRGA5VH4PDELLNU5EGA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.829627417 (625h28m49.700710452s) [noise=true , address=145.79.227.37:4567 , piece_id=EKKPOHUXXH437ZQ67K2LOPZ5773NJXLRIQTAKM7E7BTMRV5LLTJA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.829826126 (625h28m49.700909171s) [noise=true , address=145.79.227.9:4367 , piece_id=AOSHODY5MHZSG6EMAKEUDPKNDPZVULERG6NXVQWLSWHSC5UGO3IA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:49.972918433 (625h28m49.844001468s) [noise=true , address=145.79.227.5:4221 , piece_id=5XVHU4GOLP4BOEEOQ7YNLQ37LNCCBLIWIOU6BFVBROUH6X66MM7A, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.016337258 (625h28m49.887420303s) [noise=true , address=77.90.31.19:4841 , piece_id=XM72N5F4LM2GT7BFUB6KGTB7ZHBD3BKDIP2B3WQMZBRSXFTXYGUA, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.066606578 (625h28m49.937689614s) [noise=true , address=77.90.31.10:4519 , piece_id=F2UZ24G4HCUDZRFFP24SN2N2BBH7V63RWQ6PQ6ZQ476JI2GMDDAQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.802894050 (625h28m50.673977085s) [noise=true , address=45.91.116.8:4048 , piece_id=6E2UBU3KP4CCJWTALFMI6OAWBWPUOPISX7KYHOKRA5LXC2TIZJRQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: <nil> Dec 31 16:57:50.802917805 (625h28m50.67400085s) [seg_pos={0 0}, upload=548085]: Segment reached optimal threshold of 13 pieces. Dec 31 16:57:50.803136832 (625h28m50.674219877s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 31 16:57:50.803144076 (625h28m50.674227121s) [noise=true , address=77.90.31.14:4661 , piece_id=MYADO6Y4UJOLKX63WSWWW27CM7W4TL6NQRIHXYF7RBWIF2XR3QWQ, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold Dec 31 16:57:50.803253285 (625h28m50.67433633s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 31 16:57:50.803258725 (625h28m50.67434177s) [noise=true , address=45.91.116.12:4226 , piece_id=POOD3JH4PKGUPOKTW4CLUPDTRHAXP3MTQMRUDF2UB7SGBNE5KXHQ, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold Dec 31 16:57:50.803484023 (625h28m50.674567068s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 16:57:50.803502799 (625h28m50.674585845s) [noise=true , address=145.79.227.44:4195 , piece_id=3KKGZV3EZHEBN6QA2XQ4GCSHZQYRBVT5PHIZCJQYGVCHH6BTNE6Q, seg_pos={0 0}, upload=548085]: Long tail cancellation due to optimal threshold Dec 31 16:57:50.803532987 (625h28m50.674616032s) [seg_pos={0 0}, upload=548085]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 16:57:50.803658395 (625h28m50.67474145s) [upload=548085]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:57:50.803666380 (625h28m50.674749415s) [upload=548085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:57:51.091543468 (625h28m50.962626503s) [upload=548085]: Done uploading segments... Dec 31 16:57:51.091565551 (625h28m50.962648586s) [upload=548085]: Upload finished. err: <nil> Duration: 30.003673311s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [..XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.XX] duration: 5.00075153s trace: https://jaeger.dev.storj.io/trace/1165c148198ac14f FAIL (2025-12-31 16:57:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0007875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X..........X.....................................X......X] duration: 1m0.004552763s trace: https://jaeger.dev.storj.io/trace/51209bc25994253c FAIL (2025-12-31 16:57:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.882880440 (625h27m34.753963475s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.883312763 (625h27m34.754395808s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.883748182 (625h27m34.754831227s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884108909 (625h27m34.755191944s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884134819 (625h27m34.755217854s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884545450 (625h27m34.755628495s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884570638 (625h27m34.755653683s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.884987812 (625h27m34.756070847s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885002600 (625h27m34.756085635s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885364570 (625h27m34.756447615s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885437358 (625h27m34.756520394s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.885713234 (625h27m34.756796269s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886124126 (625h27m34.757207172s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886498589 (625h27m34.757581624s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:34.886844768 (625h27m34.757927803s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Uploading piece... Dec 31 16:56:36.347113954 (625h27m36.218196999s) [upload=548072]: Next returned nil segment Dec 31 16:56:36.347127068 (625h27m36.218210114s) [upload=548072]: Waiting for error group managing segments... Dec 31 16:56:36.782391943 (625h27m36.653474978s) [noise=true , address=145.79.227.2:4119 , piece_id=UBSSEQ4EILR74PC22OSCWPQOFDKPY4FT74W3X7BOFIFI5OCI23MA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.828828685 (625h27m36.69991172s) [noise=true , address=145.79.227.12:4793 , piece_id=NKCXNQYVEBULTP7UHHONXGF6ILTSUG6DZ2NCSCIFJHRYECDMVWFQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.909133386 (625h27m36.780216421s) [noise=true , address=145.79.227.7:4314 , piece_id=6L5U7UKHUMYRHPAHZP7AIFDGHNNOIMBUTBD5AYMISGOMNQFUR7CA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.951641147 (625h27m36.822724192s) [noise=true , address=77.90.31.10:4541 , piece_id=M3Y5NBJE7BBTZXFCIY3COZN7VEPSSDZA3AAIDAFKCFLSVVES2TLA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.953178845 (625h27m36.82426188s) [noise=true , address=77.90.31.16:4758 , piece_id=RQB3BR5PUIRIIUIPAH55MWGRAAFFPOKNMVHLYUX5VS7WSZVI5GJQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:36.967346992 (625h27m36.838430027s) [noise=true , address=77.90.31.10:4539 , piece_id=2UJELFLSKN6HTW6TPRVBGVYTVQAYQ64EXINWY2EDTRF62C2NYC6A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.015751392 (625h27m36.886834427s) [noise=true , address=77.90.31.12:4610 , piece_id=4W2K2M3RJ6X36C7XEBTWIGJWH34OINJJGUERDSIM5XNWDC4KTPDA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.070458612 (625h27m36.941541647s) [noise=true , address=45.91.116.12:4243 , piece_id=34VGEPVV2KHXFIR3X4VEUMX5UD4KJXZ5F573ODAMCJKF5ICIRTCQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.277203082 (625h27m37.148286118s) [noise=true , address=77.90.31.17:4781 , piece_id=7U7FMPSPICGZNPYNQIGZA4DXQLUUHXAWV3NGIHTGLOFRGVV35DZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.620497854 (625h27m37.491580909s) [noise=true , address=145.79.227.49:4362 , piece_id=EXLZZCNKQIC22WCQVZD2G5EGQGEPI5OJLBFGVAESUANK2BP2JYAQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.678713946 (625h27m37.549796991s) [noise=true , address=145.79.227.44:4213 , piece_id=6V767N4L5XZN655GRJWWTUUCNUPLEKVIW2PDEFAXUZZSFTTJQCTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.686079791 (625h27m37.557162836s) [noise=true , address=145.79.227.33:4878 , piece_id=N5PTWWSHXTUK4LBAWQ5272RLIBLSCSFKML2DSCIVQCQN7NO7KFPQ, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.719337278 (625h27m37.590420313s) [noise=true , address=145.79.227.29:4945 , piece_id=U367TT5ZN4GYR5FFCJCBZDS3FM3RZBWPGAMKFOPQSGHNZEYOTK3A, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.738954522 (625h27m37.610037557s) [noise=true , address=145.79.227.34:4939 , piece_id=4RO4RLYT2FJ2W5J2BXV5QBTRDBPEWXWTLWERVZGBXXZAPTF4MBKA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: <nil> Dec 31 16:56:37.738974741 (625h27m37.610057776s) [seg_pos={0 0}, upload=548072]: Segment reached optimal threshold of 14 pieces. Dec 31 16:56:37.739210258 (625h27m37.610293293s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rm4HueKsNdwL3eCTCSWoPseR2LQWcsE3jNPpxSfYQQorj9LL): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739218525 (625h27m37.61030157s) [noise=true , address=145.79.227.16:4931 , piece_id=MEKJOAWN564CXEFUGTUBYJNF2APXDVFV3UZGWESAKVFSVIPD6PTA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739273609 (625h27m37.610356654s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739292536 (625h27m37.610375581s) [noise=true , address=193.149.164.143:4695 , piece_id=BAS4PIQCTI4ENCDY4BHGHNLQMYRZR2PW6PTNZJZAJ3X2U3JGOJIA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739499520 (625h27m37.610582565s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF Dec 31 16:56:37.739506242 (625h27m37.610589277s) [noise=true , address=45.91.116.6:4999 , piece_id=BNUMSSY7WN7S52XJT6PWA4X7B5PG5MR5WQYH2OYBYWMVLJGPZFZA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739518896 (625h27m37.610601931s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60012->64.49.13.169:4711: use of closed network connection Dec 31 16:56:37.739527362 (625h27m37.610610407s) [noise=true , address=64.49.13.169:4711 , piece_id=TN5KR3BMTLKVEXXQVISEDFLX6B3TSQV3FMY6TEMRXEYU5GR7Z2RA, seg_pos={0 0}, upload=548072]: Long tail cancellation due to optimal threshold Dec 31 16:56:37.739545647 (625h27m37.610628692s) [seg_pos={0 0}, upload=548072]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 16:56:37.739638374 (625h27m37.610721409s) [upload=548072]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:56:37.739643183 (625h27m37.610726228s) [upload=548072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:56:38.030777232 (625h27m37.901860267s) [upload=548072]: Done uploading segments... Dec 31 16:56:38.030797912 (625h27m37.901880947s) [upload=548072]: Upload finished. err: <nil> Duration: 1m0.004563663s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [...XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.X] duration: 5.000918597s trace: https://jaeger.dev.storj.io/trace/15df357c2a777d6 FAIL (2025-12-31 16:57:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000943464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X....] duration: 6.25788509s trace: https://jaeger.dev.storj.io/trace/2357ddb7e2c2e281 FAIL (2025-12-31 16:56:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 16:54:04.303380506 (625h25m4.174463542s) [upload=548032]: Starting upload Dec 31 16:54:04.303432836 (625h25m4.174515871s) [upload=548032]: Done starting upload Dec 31 16:54:04.303545921 (625h25m4.174628976s) [upload=548032]: Uploading segments... Dec 31 16:54:04.303631374 (625h25m4.174714449s) [upload=548032]: Got next segment. Inline: false Dec 31 16:54:04.303679495 (625h25m4.17476254s) [upload=548032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 16:54:04.303700816 (625h25m4.174783851s) [upload=548032]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 16:54:04.550160820 (625h25m4.421243865s) [seg_pos={0 0}, upload=548032]: Begin upload segment... Dec 31 16:54:04.550206296 (625h25m4.421289341s) [seg_pos={0 0}, upload=548032]: Done begin upload segment. Dec 31 16:54:04.550233237 (625h25m4.421316272s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.550319773 (625h25m4.421402848s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.550895328 (625h25m4.421978363s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.551320608 (625h25m4.422403653s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.551783460 (625h25m4.422866495s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552138956 (625h25m4.423221992s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552418960 (625h25m4.423502005s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552838599 (625h25m4.423921644s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.554332483 (625h25m4.425415528s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.554476558 (625h25m4.425559593s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.555638550 (625h25m4.426721585s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.556990394 (625h25m4.428073439s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.565033940 (625h25m4.436116975s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.565184408 (625h25m4.436267443s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.566499170 (625h25m4.437582215s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.567394656 (625h25m4.438477701s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:06.264153690 (625h25m6.135236735s) [upload=548032]: Next returned nil segment Dec 31 16:54:06.264177514 (625h25m6.135260559s) [upload=548032]: Waiting for error group managing segments... Dec 31 16:54:06.536331012 (625h25m6.407414047s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.538792870 (625h25m6.409875906s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.559727663 (625h25m6.430810698s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.560250448 (625h25m6.431333493s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.569094128 (625h25m6.440177173s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.626771975 (625h25m6.49785501s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.636667477 (625h25m6.507750522s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.741721214 (625h25m6.612804259s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.863029281 (625h25m6.734112316s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.904267314 (625h25m6.775350349s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.999818748 (625h25m6.870901783s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.503007990 (625h25m7.374091025s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.956546069 (625h25m7.827629114s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.956571196 (625h25m7.827654231s) [seg_pos={0 0}, upload=548032]: Segment reached optimal threshold of 13 pieces. Dec 31 16:54:07.956812125 (625h25m7.82789517s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.956823177 (625h25m7.827906222s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957133406 (625h25m7.828216442s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.957148957 (625h25m7.828231992s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957231634 (625h25m7.828314689s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.957242886 (625h25m7.828325931s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957268374 (625h25m7.828351409s) [seg_pos={0 0}, upload=548032]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 16:54:07.957349389 (625h25m7.828432434s) [upload=548032]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:54:07.957356853 (625h25m7.828439898s) [upload=548032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:54:08.248058148 (625h25m8.119141213s) [upload=548032]: Done uploading segments... Dec 31 16:54:08.248084378 (625h25m8.119167413s) [upload=548032]: Upload finished. err: <nil> Duration: 6.257910188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..........X.....................................X......] duration: 33.863640203s trace: https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c FAIL (2025-12-31 16:56:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments... Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces. Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments... Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil> Duration: 33.863672965s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X.] duration: 3.755160296s trace: https://jaeger.dev.storj.io/trace/5e25e590cc565f32 FAIL (2025-12-31 16:56:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.755184471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X...] duration: 5.965832814s trace: https://jaeger.dev.storj.io/trace/2357ddb7e2c2e281 FAIL (2025-12-31 16:56:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 16:54:04.303380506 (625h25m4.174463542s) [upload=548032]: Starting upload Dec 31 16:54:04.303432836 (625h25m4.174515871s) [upload=548032]: Done starting upload Dec 31 16:54:04.303545921 (625h25m4.174628976s) [upload=548032]: Uploading segments... Dec 31 16:54:04.303631374 (625h25m4.174714449s) [upload=548032]: Got next segment. Inline: false Dec 31 16:54:04.303679495 (625h25m4.17476254s) [upload=548032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 16:54:04.303700816 (625h25m4.174783851s) [upload=548032]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 16:54:04.550160820 (625h25m4.421243865s) [seg_pos={0 0}, upload=548032]: Begin upload segment... Dec 31 16:54:04.550206296 (625h25m4.421289341s) [seg_pos={0 0}, upload=548032]: Done begin upload segment. Dec 31 16:54:04.550233237 (625h25m4.421316272s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.550319773 (625h25m4.421402848s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.550895328 (625h25m4.421978363s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.551320608 (625h25m4.422403653s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.551783460 (625h25m4.422866495s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552138956 (625h25m4.423221992s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552418960 (625h25m4.423502005s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552838599 (625h25m4.423921644s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.554332483 (625h25m4.425415528s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.554476558 (625h25m4.425559593s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.555638550 (625h25m4.426721585s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.556990394 (625h25m4.428073439s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.565033940 (625h25m4.436116975s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.565184408 (625h25m4.436267443s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.566499170 (625h25m4.437582215s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.567394656 (625h25m4.438477701s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:06.264153690 (625h25m6.135236735s) [upload=548032]: Next returned nil segment Dec 31 16:54:06.264177514 (625h25m6.135260559s) [upload=548032]: Waiting for error group managing segments... Dec 31 16:54:06.536331012 (625h25m6.407414047s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.538792870 (625h25m6.409875906s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.559727663 (625h25m6.430810698s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.560250448 (625h25m6.431333493s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.569094128 (625h25m6.440177173s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.626771975 (625h25m6.49785501s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.636667477 (625h25m6.507750522s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.741721214 (625h25m6.612804259s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.863029281 (625h25m6.734112316s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.904267314 (625h25m6.775350349s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.999818748 (625h25m6.870901783s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.503007990 (625h25m7.374091025s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.956546069 (625h25m7.827629114s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.956571196 (625h25m7.827654231s) [seg_pos={0 0}, upload=548032]: Segment reached optimal threshold of 13 pieces. Dec 31 16:54:07.956812125 (625h25m7.82789517s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.956823177 (625h25m7.827906222s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957133406 (625h25m7.828216442s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.957148957 (625h25m7.828231992s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957231634 (625h25m7.828314689s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.957242886 (625h25m7.828325931s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957268374 (625h25m7.828351409s) [seg_pos={0 0}, upload=548032]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 16:54:07.957349389 (625h25m7.828432434s) [upload=548032]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:54:07.957356853 (625h25m7.828439898s) [upload=548032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:54:08.248058148 (625h25m8.119141213s) [upload=548032]: Done uploading segments... Dec 31 16:54:08.248084378 (625h25m8.119167413s) [upload=548032]: Upload finished. err: <nil> Duration: 5.965877629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X..] duration: 6.073074422s trace: https://jaeger.dev.storj.io/trace/2357ddb7e2c2e281 FAIL (2025-12-31 16:55:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 16:54:04.303380506 (625h25m4.174463542s) [upload=548032]: Starting upload Dec 31 16:54:04.303432836 (625h25m4.174515871s) [upload=548032]: Done starting upload Dec 31 16:54:04.303545921 (625h25m4.174628976s) [upload=548032]: Uploading segments... Dec 31 16:54:04.303631374 (625h25m4.174714449s) [upload=548032]: Got next segment. Inline: false Dec 31 16:54:04.303679495 (625h25m4.17476254s) [upload=548032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 16:54:04.303700816 (625h25m4.174783851s) [upload=548032]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 16:54:04.550160820 (625h25m4.421243865s) [seg_pos={0 0}, upload=548032]: Begin upload segment... Dec 31 16:54:04.550206296 (625h25m4.421289341s) [seg_pos={0 0}, upload=548032]: Done begin upload segment. Dec 31 16:54:04.550233237 (625h25m4.421316272s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.550319773 (625h25m4.421402848s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.550895328 (625h25m4.421978363s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.551320608 (625h25m4.422403653s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.551783460 (625h25m4.422866495s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552138956 (625h25m4.423221992s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552418960 (625h25m4.423502005s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552838599 (625h25m4.423921644s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.554332483 (625h25m4.425415528s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.554476558 (625h25m4.425559593s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.555638550 (625h25m4.426721585s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.556990394 (625h25m4.428073439s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.565033940 (625h25m4.436116975s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.565184408 (625h25m4.436267443s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.566499170 (625h25m4.437582215s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.567394656 (625h25m4.438477701s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:06.264153690 (625h25m6.135236735s) [upload=548032]: Next returned nil segment Dec 31 16:54:06.264177514 (625h25m6.135260559s) [upload=548032]: Waiting for error group managing segments... Dec 31 16:54:06.536331012 (625h25m6.407414047s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.538792870 (625h25m6.409875906s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.559727663 (625h25m6.430810698s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.560250448 (625h25m6.431333493s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.569094128 (625h25m6.440177173s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.626771975 (625h25m6.49785501s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.636667477 (625h25m6.507750522s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.741721214 (625h25m6.612804259s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.863029281 (625h25m6.734112316s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.904267314 (625h25m6.775350349s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.999818748 (625h25m6.870901783s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.503007990 (625h25m7.374091025s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.956546069 (625h25m7.827629114s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.956571196 (625h25m7.827654231s) [seg_pos={0 0}, upload=548032]: Segment reached optimal threshold of 13 pieces. Dec 31 16:54:07.956812125 (625h25m7.82789517s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.956823177 (625h25m7.827906222s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957133406 (625h25m7.828216442s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.957148957 (625h25m7.828231992s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957231634 (625h25m7.828314689s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.957242886 (625h25m7.828325931s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957268374 (625h25m7.828351409s) [seg_pos={0 0}, upload=548032]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 16:54:07.957349389 (625h25m7.828432434s) [upload=548032]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:54:07.957356853 (625h25m7.828439898s) [upload=548032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:54:08.248058148 (625h25m8.119141213s) [upload=548032]: Done uploading segments... Dec 31 16:54:08.248084378 (625h25m8.119167413s) [upload=548032]: Upload finished. err: <nil> Duration: 6.073111202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........X.....................................X.....] duration: 38.178850496s trace: https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c FAIL (2025-12-31 16:55:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments... Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces. Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments... Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil> Duration: 38.178873461s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.X] duration: 5.002121078s trace: https://jaeger.dev.storj.io/trace/5e25e590cc565f32 FAIL (2025-12-31 16:55:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002146186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.916457127s trace: https://jaeger.dev.storj.io/trace/5582e72abe7a8819 FAIL (2025-12-31 16:55:10 UTC) orbiter warning: context deadline exceeded Duration: 6.916477025s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X.] duration: 3.538687078s trace: https://jaeger.dev.storj.io/trace/10c21ab25da0f02f FAIL (2025-12-31 16:55:04 UTC) orbiter warning: context deadline exceeded Duration: 3.538721192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..........X.....................................X....] duration: 30.58543632s trace: https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c FAIL (2025-12-31 16:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments... Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces. Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments... Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil> Duration: 30.585470122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X.] duration: 6.052573762s trace: https://jaeger.dev.storj.io/trace/2357ddb7e2c2e281 FAIL (2025-12-31 16:55:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 16:54:04.303380506 (625h25m4.174463542s) [upload=548032]: Starting upload Dec 31 16:54:04.303432836 (625h25m4.174515871s) [upload=548032]: Done starting upload Dec 31 16:54:04.303545921 (625h25m4.174628976s) [upload=548032]: Uploading segments... Dec 31 16:54:04.303631374 (625h25m4.174714449s) [upload=548032]: Got next segment. Inline: false Dec 31 16:54:04.303679495 (625h25m4.17476254s) [upload=548032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 16:54:04.303700816 (625h25m4.174783851s) [upload=548032]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 16:54:04.550160820 (625h25m4.421243865s) [seg_pos={0 0}, upload=548032]: Begin upload segment... Dec 31 16:54:04.550206296 (625h25m4.421289341s) [seg_pos={0 0}, upload=548032]: Done begin upload segment. Dec 31 16:54:04.550233237 (625h25m4.421316272s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.550319773 (625h25m4.421402848s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.550895328 (625h25m4.421978363s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.551320608 (625h25m4.422403653s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.551783460 (625h25m4.422866495s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552138956 (625h25m4.423221992s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552418960 (625h25m4.423502005s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552838599 (625h25m4.423921644s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.554332483 (625h25m4.425415528s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.554476558 (625h25m4.425559593s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.555638550 (625h25m4.426721585s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.556990394 (625h25m4.428073439s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.565033940 (625h25m4.436116975s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.565184408 (625h25m4.436267443s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.566499170 (625h25m4.437582215s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.567394656 (625h25m4.438477701s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:06.264153690 (625h25m6.135236735s) [upload=548032]: Next returned nil segment Dec 31 16:54:06.264177514 (625h25m6.135260559s) [upload=548032]: Waiting for error group managing segments... Dec 31 16:54:06.536331012 (625h25m6.407414047s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.538792870 (625h25m6.409875906s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.559727663 (625h25m6.430810698s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.560250448 (625h25m6.431333493s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.569094128 (625h25m6.440177173s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.626771975 (625h25m6.49785501s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.636667477 (625h25m6.507750522s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.741721214 (625h25m6.612804259s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.863029281 (625h25m6.734112316s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.904267314 (625h25m6.775350349s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.999818748 (625h25m6.870901783s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.503007990 (625h25m7.374091025s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.956546069 (625h25m7.827629114s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.956571196 (625h25m7.827654231s) [seg_pos={0 0}, upload=548032]: Segment reached optimal threshold of 13 pieces. Dec 31 16:54:07.956812125 (625h25m7.82789517s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.956823177 (625h25m7.827906222s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957133406 (625h25m7.828216442s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.957148957 (625h25m7.828231992s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957231634 (625h25m7.828314689s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.957242886 (625h25m7.828325931s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957268374 (625h25m7.828351409s) [seg_pos={0 0}, upload=548032]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 16:54:07.957349389 (625h25m7.828432434s) [upload=548032]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:54:07.957356853 (625h25m7.828439898s) [upload=548032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:54:08.248058148 (625h25m8.119141213s) [upload=548032]: Done uploading segments... Dec 31 16:54:08.248084378 (625h25m8.119167413s) [upload=548032]: Upload finished. err: <nil> Duration: 6.052634036s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [..X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....X] duration: 5.00092986s trace: https://jaeger.dev.storj.io/trace/10c21ab25da0f02f FAIL (2025-12-31 16:54:42 UTC) orbiter warning: context deadline exceeded Duration: 5.000951802s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.X] duration: 30.002624421s trace: https://jaeger.dev.storj.io/trace/2357ddb7e2c2e281 FAIL (2025-12-31 16:54:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 16:54:04.303380506 (625h25m4.174463542s) [upload=548032]: Starting upload Dec 31 16:54:04.303432836 (625h25m4.174515871s) [upload=548032]: Done starting upload Dec 31 16:54:04.303545921 (625h25m4.174628976s) [upload=548032]: Uploading segments... Dec 31 16:54:04.303631374 (625h25m4.174714449s) [upload=548032]: Got next segment. Inline: false Dec 31 16:54:04.303679495 (625h25m4.17476254s) [upload=548032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 16:54:04.303700816 (625h25m4.174783851s) [upload=548032]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 16:54:04.550160820 (625h25m4.421243865s) [seg_pos={0 0}, upload=548032]: Begin upload segment... Dec 31 16:54:04.550206296 (625h25m4.421289341s) [seg_pos={0 0}, upload=548032]: Done begin upload segment. Dec 31 16:54:04.550233237 (625h25m4.421316272s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.550319773 (625h25m4.421402848s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.550895328 (625h25m4.421978363s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.551320608 (625h25m4.422403653s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.551783460 (625h25m4.422866495s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552138956 (625h25m4.423221992s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552418960 (625h25m4.423502005s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.552838599 (625h25m4.423921644s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.554332483 (625h25m4.425415528s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.554476558 (625h25m4.425559593s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.555638550 (625h25m4.426721585s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.556990394 (625h25m4.428073439s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.565033940 (625h25m4.436116975s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.565184408 (625h25m4.436267443s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.566499170 (625h25m4.437582215s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:04.567394656 (625h25m4.438477701s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Uploading piece... Dec 31 16:54:06.264153690 (625h25m6.135236735s) [upload=548032]: Next returned nil segment Dec 31 16:54:06.264177514 (625h25m6.135260559s) [upload=548032]: Waiting for error group managing segments... Dec 31 16:54:06.536331012 (625h25m6.407414047s) [noise=true , address=145.79.227.52:4445 , piece_id=KLVNKKAVLTCDXBPGQDFLEOXOODU2KS4MPO4Y6OEPSVMX4AXRRWEQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.538792870 (625h25m6.409875906s) [noise=true , address=145.79.227.15:4874 , piece_id=3CVWADJWSRXY6B5SR5NCXK2ROZKQDUWX66NPHV3B3HN3AC72666Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.559727663 (625h25m6.430810698s) [noise=true , address=145.79.227.4:4185 , piece_id=RI6CLWZ2ZXPQ36SWRQTRI3ECAFTSHUK5TWFKYYXTE7CERSD3YGIQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.560250448 (625h25m6.431333493s) [noise=true , address=145.79.227.9:4367 , piece_id=MGWDE3UGN7OTFEHKEZKF3CTD4WMZWOXYIKU7543ZXQAXHHBQTI7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.569094128 (625h25m6.440177173s) [noise=true , address=145.79.227.14:4837 , piece_id=JHFAKK2Q6GUN3LSB7OLZE4QH5BQ3V2ANPT4QG6FQ7XC3SYPXSFLQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.626771975 (625h25m6.49785501s) [noise=true , address=145.79.227.38:4977 , piece_id=LZHY2WNZ6X23TN65R4PVJWLDYV4K57BQ4C6LRPXHVYNYIEO57S7Q, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.636667477 (625h25m6.507750522s) [noise=true , address=145.79.227.19:4079 , piece_id=TYQFG66ZLQRS5OTMLNOYRR24B7HF6FJKEBOZNZK2EO4I3PIU65AQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.741721214 (625h25m6.612804259s) [noise=true , address=77.90.31.10:4517 , piece_id=NDSDOK2KXTFHPZ237OIQOEKEWR5PAEJKN6L4RFXSYNHFXISGYHPA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.863029281 (625h25m6.734112316s) [noise=true , address=45.91.116.13:4153 , piece_id=MNIXKBZDEUHYK2Q66CVJFJJDNJMRICPWD4QVUZNKBOVSMKXIW4EA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.904267314 (625h25m6.775350349s) [noise=true , address=77.90.31.15:4697 , piece_id=MNW477DNTP25EHJMHVVW3BX3UYDBJGG7FXHFSTFZLTM2ZW4NV4ZQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:06.999818748 (625h25m6.870901783s) [noise=true , address=45.91.116.14:4191 , piece_id=HSXBPP3II2CHJHX3BFXR4BNCTAQDGLIRGBAPGHV7ZUP5IDMBTDQA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.503007990 (625h25m7.374091025s) [noise=true , address=77.90.31.14:4662 , piece_id=Y3JXTJ3RYGB2JNDSRGS4XDA5BOVOS6WEGK44HZLYYLDTPOQXXQTA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.956546069 (625h25m7.827629114s) [noise=true , address=45.91.116.8:4046 , piece_id=YKXTIPM2QQD45UADUYGAUXH4TQFJII3SUKJ3W5NMYJYLNF4O57TA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: <nil> Dec 31 16:54:07.956571196 (625h25m7.827654231s) [seg_pos={0 0}, upload=548032]: Segment reached optimal threshold of 13 pieces. Dec 31 16:54:07.956812125 (625h25m7.82789517s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.956823177 (625h25m7.827906222s) [noise=true , address=77.90.31.21:4877 , piece_id=UBDL75FSPJS6UT62FA3REOMF5EC3TJWYFSL6S3BMGAAUN5PRCMGA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957133406 (625h25m7.828216442s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.957148957 (625h25m7.828231992s) [noise=true , address=77.90.31.12:4589 , piece_id=6ZVQXJL5EGFM5735LNMAMOZVDKE7WQQLQ7QLAUY4ZAMDUNFVVUXA, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957231634 (625h25m7.828314689s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 31 16:54:07.957242886 (625h25m7.828325931s) [noise=true , address=45.91.116.10:4082 , piece_id=QOJDQDWLNJFAIVOV7JOSGW4RD5FPU7NAZE3OIKIHHV2245LNY7VQ, seg_pos={0 0}, upload=548032]: Long tail cancellation due to optimal threshold Dec 31 16:54:07.957268374 (625h25m7.828351409s) [seg_pos={0 0}, upload=548032]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 16:54:07.957349389 (625h25m7.828432434s) [upload=548032]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:54:07.957356853 (625h25m7.828439898s) [upload=548032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:54:08.248058148 (625h25m8.119141213s) [upload=548032]: Done uploading segments... Dec 31 16:54:08.248084378 (625h25m8.119167413s) [upload=548032]: Upload finished. err: <nil> Duration: 30.002645482s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.938244412s trace: https://jaeger.dev.storj.io/trace/5582e72abe7a8819 FAIL (2025-12-31 16:54:18 UTC) orbiter warning: context deadline exceeded Duration: 5.938266224s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..........X.....................................X...] duration: 44.945188361s trace: https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c FAIL (2025-12-31 16:54:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments... Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces. Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments... Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil> Duration: 44.945203308s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [...X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX....] duration: 3.533593772s trace: https://jaeger.dev.storj.io/trace/19c6aa42963b3c0f FAIL (2025-12-31 16:53:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.53362424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX..XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX.] duration: 8.095222294s trace: https://jaeger.dev.storj.io/trace/5cfa15b3395c99c7 FAIL (2025-12-31 16:53:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA) context deadline exceeded ======LOGS====== (625h24m0.187935032s) [upload=548018]: Uploading segments... Dec 31 16:53:00.317161005 (625h24m0.18824405s) [upload=548018]: Got next segment. Inline: false Dec 31 16:53:00.317208526 (625h24m0.188291572s) [upload=548018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 16:53:00.317214959 (625h24m0.188298004s) [upload=548018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 16:53:00.559124607 (625h24m0.430207652s) [seg_pos={0 0}, upload=548018]: Begin upload segment... Dec 31 16:53:00.559295773 (625h24m0.430378808s) [seg_pos={0 0}, upload=548018]: Done begin upload segment. Dec 31 16:53:00.559331241 (625h24m0.430414276s) [noise=true , address=77.90.31.14:4663 , piece_id=HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.560814085 (625h24m0.43189712s) [noise=true , address=145.79.227.22:4511 , piece_id=COUKMWSBDOUPUV6IBMTMM7MDRLJFR2QVHI4P6TD6K7XXLDKTID5Q, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.561408707 (625h24m0.432491752s) [noise=true , address=145.79.227.8:4330 , piece_id=XG67YYMPMX3ICYAE7AGFG4EFFIQILXFPEL563MCXVXCFX243LKBQ, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.561720671 (625h24m0.432803716s) [noise=true , address=145.79.227.41:4087 , piece_id=2NMUN4BSMAGHN4VFZM3AEQWHZN6S2IGMKSURL5EKEPB4BIIUAZFA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.561960947 (625h24m0.433043982s) [noise=true , address=145.79.227.52:4446 , piece_id=RQCOSCULWHAOXN662DD7G5OYXFWYWF7K6ZASKM3DFJO35WTGBWAA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.562345741 (625h24m0.433428786s) [noise=true , address=45.91.116.7:4012 , piece_id=NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.562382371 (625h24m0.433465416s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.562671952 (625h24m0.433754997s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.562940604 (625h24m0.434023649s) [noise=true , address=145.79.227.39:4015 , piece_id=E3NXLXBM3IIK7ICGMIOGAZOGUVE24U4O3S5TLV5OFJAGD6NRGJVA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.562982724 (625h24m0.434065799s) [noise=true , address=45.91.116.10:4082 , piece_id=72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.563555213 (625h24m0.434638258s) [noise=true , address=145.79.227.38:4978 , piece_id=BOVSXWQR7323NSG3D2T2RNWG25TYTRQQ7C4RCP7RVEOM65HXFPLQ, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.564216943 (625h24m0.435299988s) [noise=true , address=145.79.227.42:4123 , piece_id=2KOMXT4FVOLDWYLFFTOT3Z7QRC2SQPUMY3WJROPBAS6EV7FNSTIA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.564915253 (625h24m0.435998288s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.565695830 (625h24m0.436778875s) [noise=true , address=145.79.227.32:4835 , piece_id=ZUMYIO55NMELJXSHOJSDXGEMGFEXYTAOFTQUCAWL2CW4UFZFRHMQ, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.565999679 (625h24m0.437082714s) [noise=true , address=145.79.227.3:4150 , piece_id=IICUES4EAAPXXEUZHMS5M742ATNEZ6ZH2L6343NBEFWKN5A4EDTA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.566426211 (625h24m0.437509246s) [noise=true , address=145.79.227.20:4437 , piece_id=ZB5POSRWAFSXFQBHGP6BNWH7FV6SHBUNWPB5M2GBGHFEZ72P5P6A, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:02.269582063 (625h24m2.140665109s) [upload=548018]: Next returned nil segment Dec 31 16:53:02.269601401 (625h24m2.140684456s) [upload=548018]: Waiting for error group managing segments... Dec 31 16:53:02.550702465 (625h24m2.42178551s) [noise=true , address=145.79.227.20:4437 , piece_id=ZB5POSRWAFSXFQBHGP6BNWH7FV6SHBUNWPB5M2GBGHFEZ72P5P6A, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.554501710 (625h24m2.425584765s) [noise=true , address=145.79.227.52:4446 , piece_id=RQCOSCULWHAOXN662DD7G5OYXFWYWF7K6ZASKM3DFJO35WTGBWAA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.569282905 (625h24m2.44036595s) [noise=true , address=145.79.227.3:4150 , piece_id=IICUES4EAAPXXEUZHMS5M742ATNEZ6ZH2L6343NBEFWKN5A4EDTA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.572389480 (625h24m2.443472526s) [noise=true , address=145.79.227.22:4511 , piece_id=COUKMWSBDOUPUV6IBMTMM7MDRLJFR2QVHI4P6TD6K7XXLDKTID5Q, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.579231618 (625h24m2.450314664s) [noise=true , address=145.79.227.8:4330 , piece_id=XG67YYMPMX3ICYAE7AGFG4EFFIQILXFPEL563MCXVXCFX243LKBQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.587215912 (625h24m2.458298957s) [noise=true , address=145.79.227.32:4835 , piece_id=ZUMYIO55NMELJXSHOJSDXGEMGFEXYTAOFTQUCAWL2CW4UFZFRHMQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.648133958 (625h24m2.519217003s) [noise=true , address=145.79.227.41:4087 , piece_id=2NMUN4BSMAGHN4VFZM3AEQWHZN6S2IGMKSURL5EKEPB4BIIUAZFA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.651541286 (625h24m2.522624331s) [noise=true , address=145.79.227.39:4015 , piece_id=E3NXLXBM3IIK7ICGMIOGAZOGUVE24U4O3S5TLV5OFJAGD6NRGJVA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.708542554 (625h24m2.579625599s) [noise=true , address=145.79.227.38:4978 , piece_id=BOVSXWQR7323NSG3D2T2RNWG25TYTRQQ7C4RCP7RVEOM65HXFPLQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.724187364 (625h24m2.595270409s) [noise=true , address=145.79.227.42:4123 , piece_id=2KOMXT4FVOLDWYLFFTOT3Z7QRC2SQPUMY3WJROPBAS6EV7FNSTIA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.726115565 (625h24m2.5971986s) [noise=true , address=77.90.31.14:4663 , piece_id=HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.897950584 (625h24m2.769033629s) [noise=true , address=45.91.116.10:4082 , piece_id=72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.988832015 (625h24m2.85991506s) [noise=true , address=45.91.116.7:4012 , piece_id=NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.988854056 (625h24m2.859937101s) [seg_pos={0 0}, upload=548018]: Segment reached optimal threshold of 13 pieces. Dec 31 16:53:02.989090487 (625h24m2.860173532s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection Dec 31 16:53:02.989097470 (625h24m2.860180515s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold Dec 31 16:53:02.989284146 (625h24m2.860367191s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 31 16:53:02.989301869 (625h24m2.860384914s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold Dec 31 16:53:02.989316708 (625h24m2.860399743s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection Dec 31 16:53:02.989324032 (625h24m2.860407077s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold Dec 31 16:53:02.989342877 (625h24m2.860425922s) [seg_pos={0 0}, upload=548018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 16:53:02.989422960 (625h24m2.860506015s) [upload=548018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:53:02.989428981 (625h24m2.860512037s) [upload=548018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:53:03.276557574 (625h24m3.147640619s) [upload=548018]: Done uploading segments... Dec 31 16:53:03.276584946 (625h24m3.147667991s) [upload=548018]: Upload finished. err: <nil> Duration: 8.095252351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX..XX........X....X.X...X....X.X.......X.XX..X..X...X.X.XX] duration: 30.003693707s trace: https://jaeger.dev.storj.io/trace/5cfa15b3395c99c7 FAIL (2025-12-31 16:53:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA) context deadline exceeded ======LOGS====== (625h24m0.187935032s) [upload=548018]: Uploading segments... Dec 31 16:53:00.317161005 (625h24m0.18824405s) [upload=548018]: Got next segment. Inline: false Dec 31 16:53:00.317208526 (625h24m0.188291572s) [upload=548018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 16:53:00.317214959 (625h24m0.188298004s) [upload=548018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 16:53:00.559124607 (625h24m0.430207652s) [seg_pos={0 0}, upload=548018]: Begin upload segment... Dec 31 16:53:00.559295773 (625h24m0.430378808s) [seg_pos={0 0}, upload=548018]: Done begin upload segment. Dec 31 16:53:00.559331241 (625h24m0.430414276s) [noise=true , address=77.90.31.14:4663 , piece_id=HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.560814085 (625h24m0.43189712s) [noise=true , address=145.79.227.22:4511 , piece_id=COUKMWSBDOUPUV6IBMTMM7MDRLJFR2QVHI4P6TD6K7XXLDKTID5Q, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.561408707 (625h24m0.432491752s) [noise=true , address=145.79.227.8:4330 , piece_id=XG67YYMPMX3ICYAE7AGFG4EFFIQILXFPEL563MCXVXCFX243LKBQ, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.561720671 (625h24m0.432803716s) [noise=true , address=145.79.227.41:4087 , piece_id=2NMUN4BSMAGHN4VFZM3AEQWHZN6S2IGMKSURL5EKEPB4BIIUAZFA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.561960947 (625h24m0.433043982s) [noise=true , address=145.79.227.52:4446 , piece_id=RQCOSCULWHAOXN662DD7G5OYXFWYWF7K6ZASKM3DFJO35WTGBWAA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.562345741 (625h24m0.433428786s) [noise=true , address=45.91.116.7:4012 , piece_id=NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.562382371 (625h24m0.433465416s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.562671952 (625h24m0.433754997s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.562940604 (625h24m0.434023649s) [noise=true , address=145.79.227.39:4015 , piece_id=E3NXLXBM3IIK7ICGMIOGAZOGUVE24U4O3S5TLV5OFJAGD6NRGJVA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.562982724 (625h24m0.434065799s) [noise=true , address=45.91.116.10:4082 , piece_id=72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.563555213 (625h24m0.434638258s) [noise=true , address=145.79.227.38:4978 , piece_id=BOVSXWQR7323NSG3D2T2RNWG25TYTRQQ7C4RCP7RVEOM65HXFPLQ, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.564216943 (625h24m0.435299988s) [noise=true , address=145.79.227.42:4123 , piece_id=2KOMXT4FVOLDWYLFFTOT3Z7QRC2SQPUMY3WJROPBAS6EV7FNSTIA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.564915253 (625h24m0.435998288s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.565695830 (625h24m0.436778875s) [noise=true , address=145.79.227.32:4835 , piece_id=ZUMYIO55NMELJXSHOJSDXGEMGFEXYTAOFTQUCAWL2CW4UFZFRHMQ, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.565999679 (625h24m0.437082714s) [noise=true , address=145.79.227.3:4150 , piece_id=IICUES4EAAPXXEUZHMS5M742ATNEZ6ZH2L6343NBEFWKN5A4EDTA, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:00.566426211 (625h24m0.437509246s) [noise=true , address=145.79.227.20:4437 , piece_id=ZB5POSRWAFSXFQBHGP6BNWH7FV6SHBUNWPB5M2GBGHFEZ72P5P6A, seg_pos={0 0}, upload=548018]: Uploading piece... Dec 31 16:53:02.269582063 (625h24m2.140665109s) [upload=548018]: Next returned nil segment Dec 31 16:53:02.269601401 (625h24m2.140684456s) [upload=548018]: Waiting for error group managing segments... Dec 31 16:53:02.550702465 (625h24m2.42178551s) [noise=true , address=145.79.227.20:4437 , piece_id=ZB5POSRWAFSXFQBHGP6BNWH7FV6SHBUNWPB5M2GBGHFEZ72P5P6A, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.554501710 (625h24m2.425584765s) [noise=true , address=145.79.227.52:4446 , piece_id=RQCOSCULWHAOXN662DD7G5OYXFWYWF7K6ZASKM3DFJO35WTGBWAA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.569282905 (625h24m2.44036595s) [noise=true , address=145.79.227.3:4150 , piece_id=IICUES4EAAPXXEUZHMS5M742ATNEZ6ZH2L6343NBEFWKN5A4EDTA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.572389480 (625h24m2.443472526s) [noise=true , address=145.79.227.22:4511 , piece_id=COUKMWSBDOUPUV6IBMTMM7MDRLJFR2QVHI4P6TD6K7XXLDKTID5Q, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.579231618 (625h24m2.450314664s) [noise=true , address=145.79.227.8:4330 , piece_id=XG67YYMPMX3ICYAE7AGFG4EFFIQILXFPEL563MCXVXCFX243LKBQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.587215912 (625h24m2.458298957s) [noise=true , address=145.79.227.32:4835 , piece_id=ZUMYIO55NMELJXSHOJSDXGEMGFEXYTAOFTQUCAWL2CW4UFZFRHMQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.648133958 (625h24m2.519217003s) [noise=true , address=145.79.227.41:4087 , piece_id=2NMUN4BSMAGHN4VFZM3AEQWHZN6S2IGMKSURL5EKEPB4BIIUAZFA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.651541286 (625h24m2.522624331s) [noise=true , address=145.79.227.39:4015 , piece_id=E3NXLXBM3IIK7ICGMIOGAZOGUVE24U4O3S5TLV5OFJAGD6NRGJVA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.708542554 (625h24m2.579625599s) [noise=true , address=145.79.227.38:4978 , piece_id=BOVSXWQR7323NSG3D2T2RNWG25TYTRQQ7C4RCP7RVEOM65HXFPLQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.724187364 (625h24m2.595270409s) [noise=true , address=145.79.227.42:4123 , piece_id=2KOMXT4FVOLDWYLFFTOT3Z7QRC2SQPUMY3WJROPBAS6EV7FNSTIA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.726115565 (625h24m2.5971986s) [noise=true , address=77.90.31.14:4663 , piece_id=HVYJJTDKP2PNFD4EPFN3NV5O3EPMD5REU3VF5KPAOZRMSSL3REQQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.897950584 (625h24m2.769033629s) [noise=true , address=45.91.116.10:4082 , piece_id=72ZVQSHOUQHKQUBCGG3UZDQWLLDSTCHQR7RRZYARVWSMVVGWPEYA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.988832015 (625h24m2.85991506s) [noise=true , address=45.91.116.7:4012 , piece_id=NZYM32SUJY5H5FVGJQL3ZBQZNGVJVBQMU7O4SRZX6RXPBK34NRFA, seg_pos={0 0}, upload=548018]: Done uploading piece. err: <nil> Dec 31 16:53:02.988854056 (625h24m2.859937101s) [seg_pos={0 0}, upload=548018]: Segment reached optimal threshold of 13 pieces. Dec 31 16:53:02.989090487 (625h24m2.860173532s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43678->45.91.116.11:4118: use of closed network connection Dec 31 16:53:02.989097470 (625h24m2.860180515s) [noise=true , address=45.91.116.11:4118 , piece_id=2J6BXGMDT4EHSNS5IPGT4RUOFKFLJ56OFV4HUJUJA2SBYJ3JBK2Q, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold Dec 31 16:53:02.989284146 (625h24m2.860367191s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 31 16:53:02.989301869 (625h24m2.860384914s) [noise=true , address=45.91.116.13:4154 , piece_id=HUUYW6JZ6RQHLX6K5QOPNOLA4WHR2QJZBLF6XAU2KG2GHIIXU67A, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold Dec 31 16:53:02.989316708 (625h24m2.860399743s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34968->45.91.116.9:4261: use of closed network connection Dec 31 16:53:02.989324032 (625h24m2.860407077s) [noise=true , address=45.91.116.9:4261 , piece_id=DKGELPLYLSA46I6OHE4K3NNLIJLRB66DNFCQK7HVYIPP54L5RZGQ, seg_pos={0 0}, upload=548018]: Long tail cancellation due to optimal threshold Dec 31 16:53:02.989342877 (625h24m2.860425922s) [seg_pos={0 0}, upload=548018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 16:53:02.989422960 (625h24m2.860506015s) [upload=548018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:53:02.989428981 (625h24m2.860512037s) [upload=548018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:53:03.276557574 (625h24m3.147640619s) [upload=548018]: Done uploading segments... Dec 31 16:53:03.276584946 (625h24m3.147667991s) [upload=548018]: Upload finished. err: <nil> Duration: 30.003720928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..........X.....................................X..] duration: 33.657750366s trace: https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c FAIL (2025-12-31 16:53:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments... Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces. Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments... Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil> Duration: 33.657777237s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [X...X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX...] duration: 3.607174747s trace: https://jaeger.dev.storj.io/trace/19c6aa42963b3c0f FAIL (2025-12-31 16:53:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.60719771s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.510578118s trace: https://jaeger.dev.storj.io/trace/5582e72abe7a8819 FAIL (2025-12-31 16:53:05 UTC) orbiter warning: context deadline exceeded Duration: 6.510598367s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........X.....................................X.] duration: 30.996530897s trace: https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c FAIL (2025-12-31 16:52:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments... Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces. Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments... Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil> Duration: 30.996561116s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX..XX........X....X.X...X....X.X.......X.XX..X..X...X.X.X] duration: 30.002644045s trace: https://jaeger.dev.storj.io/trace/3df2b37f6c869d10 FAIL (2025-12-31 16:52:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: RJ5LFCAVPJPYGDEQ5UE6GM2Y3BKZMAZUB6SCZB2FKQ6ZMXHXTAEQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: EOKR653TKNQIX2NOGXBKC2XYBNDNKTMSK352BDRQNPQ3QZGJE7ZA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: S27E2P53OPIFMITM45LLMRWF4YEJE3JHE3REC3VJB353SDHJ2YXQ) context deadline exceeded ======LOGS====== Dec 31 16:52:07.448440614 (625h23m7.319523649s) [upload=548003]: Starting upload Dec 31 16:52:07.448534873 (625h23m7.319617918s) [upload=548003]: Done starting upload Dec 31 16:52:07.448605697 (625h23m7.319688732s) [upload=548003]: Uploading segments... Dec 31 16:52:07.448830025 (625h23m7.31991307s) [upload=548003]: Got next segment. Inline: false Dec 31 16:52:07.449025136 (625h23m7.320108171s) [upload=548003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 16:52:07.449033202 (625h23m7.320116247s) [upload=548003]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 16:52:07.696415663 (625h23m7.567498708s) [seg_pos={0 0}, upload=548003]: Begin upload segment... Dec 31 16:52:07.696478633 (625h23m7.567561678s) [seg_pos={0 0}, upload=548003]: Done begin upload segment. Dec 31 16:52:07.696500764 (625h23m7.567583809s) [noise=true , address=145.79.227.20:4438 , piece_id=J3OGLW3BAROIWAQVC36W47NB5UIYR44Y57BHS7S5KAVVYIXKES3Q, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.696522135 (625h23m7.56760517s) [noise=true , address=45.91.116.14:4192 , piece_id=JPDOMSEQNR4676DVMOPZPRUEM2RBW55BJJHEJJ4HIRYE7FLS7E7A, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.697181461 (625h23m7.568264506s) [noise=true , address=45.91.116.9:4263 , piece_id=S27E2P53OPIFMITM45LLMRWF4YEJE3JHE3REC3VJB353SDHJ2YXQ, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.697585610 (625h23m7.568668645s) [noise=true , address=77.90.31.21:4880 , piece_id=RJ5LFCAVPJPYGDEQ5UE6GM2Y3BKZMAZUB6SCZB2FKQ6ZMXHXTAEQ, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.697991964 (625h23m7.569075009s) [noise=true , address=145.79.227.48:4303 , piece_id=M76HKAHB66KPBIHSQU6YLE3UEZT2FDASTZOOQTHELVW7O42ED5TA, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.698363962 (625h23m7.569447008s) [noise=true , address=145.79.227.10:4401 , piece_id=VY5VYDQ27J6GZT2DJMY2BWASD3SN6F3JOMTLACLUWWNTT7WWSZPA, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.698727164 (625h23m7.569810209s) [noise=true , address=145.79.227.43:4158 , piece_id=MKTWCS7FKV6FJSKTEPT7OCBJNZPD6ZGM6LGY7Q3J3MUPXQGRKVOQ, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.699132686 (625h23m7.570215731s) [noise=true , address=77.90.31.17:4773 , piece_id=LSRB5SXJEOV7RQXKXKDXZ5TP5H36LTFGXMJBGVSH6PIYGDAETKTQ, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.699581771 (625h23m7.570664806s) [noise=true , address=145.79.227.25:4619 , piece_id=TSG6P6HGSDQ3SJ762SXTXKTUQE2RKV5MW7LX77V6P3IWV6ED3C3A, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.700034804 (625h23m7.571117849s) [noise=true , address=145.79.227.52:4445 , piece_id=NA3LOZY6RHDC45GI76UXXM4F6QN53XRNR3UVVRAHYVHMLHQ4EUMQ, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.701074304 (625h23m7.572157339s) [noise=true , address=145.79.227.2:4115 , piece_id=5T5PKDZ6MUKWQ7C54A26HESEDEZ6PYD6YSGSCEWK7R2AFDKLUYMQ, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.701180105 (625h23m7.57226314s) [noise=true , address=145.79.227.16:4912 , piece_id=5G3PEN6VZTWDG36SDVFWB6IH2KRTT4M4WXMOUIM65NXZKQW4VZYA, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.702344933 (625h23m7.573427978s) [noise=true , address=145.79.227.27:4691 , piece_id=CBSV5SVEI4G2QRC6YRVVBUX27TBMIK7DCSBELL2WMZFMRCXO662Q, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.702614065 (625h23m7.57369711s) [noise=true , address=45.91.116.12:4228 , piece_id=ASSF452HMF3XCCXWXZP3HQN5SM7ORYWVNAXAW4XLR2M62HPW2GQQ, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.703533977 (625h23m7.574617012s) [noise=true , address=145.79.227.9:4367 , piece_id=JHVU6SSS7BZ5BFI2O5VLYHAU6RVXS7NMKQISS2465N7UIVDMWI5Q, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:07.703906065 (625h23m7.57498911s) [noise=true , address=45.91.116.6:4973 , piece_id=EOKR653TKNQIX2NOGXBKC2XYBNDNKTMSK352BDRQNPQ3QZGJE7ZA, seg_pos={0 0}, upload=548003]: Uploading piece... Dec 31 16:52:09.392753555 (625h23m9.2638366s) [upload=548003]: Next returned nil segment Dec 31 16:52:09.392793131 (625h23m9.263876176s) [upload=548003]: Waiting for error group managing segments... Dec 31 16:52:09.671552467 (625h23m9.542635502s) [noise=true , address=145.79.227.10:4401 , piece_id=VY5VYDQ27J6GZT2DJMY2BWASD3SN6F3JOMTLACLUWWNTT7WWSZPA, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:09.688655704 (625h23m9.559738749s) [noise=true , address=145.79.227.25:4619 , piece_id=TSG6P6HGSDQ3SJ762SXTXKTUQE2RKV5MW7LX77V6P3IWV6ED3C3A, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:09.693405949 (625h23m9.564488984s) [noise=true , address=145.79.227.20:4438 , piece_id=J3OGLW3BAROIWAQVC36W47NB5UIYR44Y57BHS7S5KAVVYIXKES3Q, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:09.699196074 (625h23m9.570279119s) [noise=true , address=145.79.227.27:4691 , piece_id=CBSV5SVEI4G2QRC6YRVVBUX27TBMIK7DCSBELL2WMZFMRCXO662Q, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:09.702592241 (625h23m9.573675286s) [noise=true , address=145.79.227.52:4445 , piece_id=NA3LOZY6RHDC45GI76UXXM4F6QN53XRNR3UVVRAHYVHMLHQ4EUMQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:09.710118462 (625h23m9.581201507s) [noise=true , address=145.79.227.48:4303 , piece_id=M76HKAHB66KPBIHSQU6YLE3UEZT2FDASTZOOQTHELVW7O42ED5TA, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:09.717270661 (625h23m9.588353696s) [noise=true , address=145.79.227.16:4912 , piece_id=5G3PEN6VZTWDG36SDVFWB6IH2KRTT4M4WXMOUIM65NXZKQW4VZYA, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:09.762149617 (625h23m9.633232662s) [noise=true , address=145.79.227.9:4367 , piece_id=JHVU6SSS7BZ5BFI2O5VLYHAU6RVXS7NMKQISS2465N7UIVDMWI5Q, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:09.961121327 (625h23m9.832204372s) [noise=true , address=77.90.31.21:4880 , piece_id=RJ5LFCAVPJPYGDEQ5UE6GM2Y3BKZMAZUB6SCZB2FKQ6ZMXHXTAEQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:10.106798796 (625h23m9.977881831s) [noise=true , address=45.91.116.12:4228 , piece_id=ASSF452HMF3XCCXWXZP3HQN5SM7ORYWVNAXAW4XLR2M62HPW2GQQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:10.144953117 (625h23m10.016036152s) [noise=true , address=45.91.116.14:4192 , piece_id=JPDOMSEQNR4676DVMOPZPRUEM2RBW55BJJHEJJ4HIRYE7FLS7E7A, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:10.301599614 (625h23m10.172682649s) [noise=true , address=45.91.116.6:4973 , piece_id=EOKR653TKNQIX2NOGXBKC2XYBNDNKTMSK352BDRQNPQ3QZGJE7ZA, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:10.582860483 (625h23m10.453943518s) [noise=true , address=45.91.116.9:4263 , piece_id=S27E2P53OPIFMITM45LLMRWF4YEJE3JHE3REC3VJB353SDHJ2YXQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: <nil> Dec 31 16:52:10.582885370 (625h23m10.453968405s) [seg_pos={0 0}, upload=548003]: Segment reached optimal threshold of 13 pieces. Dec 31 16:52:10.583123755 (625h23m10.4542068s) [noise=true , address=145.79.227.43:4158 , piece_id=MKTWCS7FKV6FJSKTEPT7OCBJNZPD6ZGM6LGY7Q3J3MUPXQGRKVOQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 31 16:52:10.583132952 (625h23m10.454215997s) [noise=true , address=145.79.227.43:4158 , piece_id=MKTWCS7FKV6FJSKTEPT7OCBJNZPD6ZGM6LGY7Q3J3MUPXQGRKVOQ, seg_pos={0 0}, upload=548003]: Long tail cancellation due to optimal threshold Dec 31 16:52:10.583320418 (625h23m10.454403463s) [noise=true , address=145.79.227.2:4115 , piece_id=5T5PKDZ6MUKWQ7C54A26HESEDEZ6PYD6YSGSCEWK7R2AFDKLUYMQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42976->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:42976->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42976->145.79.227.2:4115: use of closed network connection Dec 31 16:52:10.583339255 (625h23m10.4544223s) [noise=true , address=145.79.227.2:4115 , piece_id=5T5PKDZ6MUKWQ7C54A26HESEDEZ6PYD6YSGSCEWK7R2AFDKLUYMQ, seg_pos={0 0}, upload=548003]: Long tail cancellation due to optimal threshold Dec 31 16:52:10.583448493 (625h23m10.454531538s) [noise=true , address=77.90.31.17:4773 , piece_id=LSRB5SXJEOV7RQXKXKDXZ5TP5H36LTFGXMJBGVSH6PIYGDAETKTQ, seg_pos={0 0}, upload=548003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 31 16:52:10.583458581 (625h23m10.454541627s) [noise=true , address=77.90.31.17:4773 , piece_id=LSRB5SXJEOV7RQXKXKDXZ5TP5H36LTFGXMJBGVSH6PIYGDAETKTQ, seg_pos={0 0}, upload=548003]: Long tail cancellation due to optimal threshold Dec 31 16:52:10.583485132 (625h23m10.454568177s) [seg_pos={0 0}, upload=548003]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 16:52:10.583567569 (625h23m10.454650604s) [upload=548003]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:52:10.583575395 (625h23m10.45465843s) [upload=548003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:52:10.880724708 (625h23m10.751807763s) [upload=548003]: Done uploading segments... Dec 31 16:52:10.880748953 (625h23m10.751831999s) [upload=548003]: Upload finished. err: <nil> Duration: 30.002668311s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.X...X....XXXXXXXXX.XXX.........XXXX..XXXXXX.X.XXX.X.X.XXX..] duration: 4.164442846s trace: https://jaeger.dev.storj.io/trace/19c6aa42963b3c0f FAIL (2025-12-31 16:52:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.164470418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........X.....................................X] duration: 1m0.002952776s trace: https://jaeger.dev.storj.io/trace/1fe9f44dea5d609c FAIL (2025-12-31 16:52:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .241246563 (625h22m16.112329598s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.241981543 (625h22m16.113064588s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242111701 (625h22m16.113194746s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242433975 (625h22m16.11351702s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242713457 (625h22m16.113796492s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.242802286 (625h22m16.113885331s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243245350 (625h22m16.114328395s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.243631865 (625h22m16.114714901s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244008643 (625h22m16.115091678s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244381102 (625h22m16.115464147s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.244758581 (625h22m16.115841616s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245176256 (625h22m16.116259301s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245538285 (625h22m16.11662133s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.245960649 (625h22m16.117043684s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.246337136 (625h22m16.117420181s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247466537 (625h22m16.118549572s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:16.247996927 (625h22m16.119079962s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Uploading piece... Dec 31 16:51:17.693671241 (625h22m17.564754286s) [upload=547992]: Next returned nil segment Dec 31 16:51:17.693714864 (625h22m17.564797899s) [upload=547992]: Waiting for error group managing segments... Dec 31 16:51:18.120111632 (625h22m17.991194667s) [noise=true , address=145.79.227.19:4096 , piece_id=LRU5MNH2XCDGAU2SM6ZKNIJ62YEREM3YNQUR6WUFD5UCKVXDLNBQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.122023212 (625h22m17.993106257s) [noise=true , address=145.79.227.13:4829 , piece_id=XPRY7LTSSVPAWZ5IZX7HGWIV3BPVTCBVDJSRYCE3O5V24JLATOXQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.201357746 (625h22m18.072440781s) [noise=true , address=145.79.227.39:4017 , piece_id=FEX6B2LSN7DTOM2MY6SAUCQBLRNXPUYJQCIL4INC5ZI2YMFPO2YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.293128900 (625h22m18.164211965s) [noise=true , address=77.90.31.18:4812 , piece_id=6SQFD5RFZU2BIT7ECWGIBC55TKW5XE3BGR6FBXKZJKRVK2VFL4WA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.306933417 (625h22m18.178016452s) [noise=true , address=77.90.31.15:4727 , piece_id=MJFOZ7ZU7Q55EDGUQ2XV5VGVY7FOVVA4XVRTAP6C3HBF4UD5TM5Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.357131932 (625h22m18.228214977s) [noise=true , address=77.90.31.11:4563 , piece_id=2LRC66FCB4DTXU44NFN5O5L6ZMGDCQ6LHHB7TLYJ2LZSYKRWJEYQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.390427070 (625h22m18.261510105s) [noise=true , address=77.90.31.14:4685 , piece_id=TLIWWP7RHAKUZABAVJ5SASRTGPQVZ2W5B4NKL6W2JM5G6X2Z7SIA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.469358907 (625h22m18.340441942s) [noise=true , address=45.91.116.10:4086 , piece_id=QFOZ2ETB4MNHFF3SHNEWU32HZELNGHK7GSNRZ4EWUZ47P3KKGLZQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.675586202 (625h22m18.546669267s) [noise=true , address=77.90.31.16:4768 , piece_id=OILJHWPLBSG5O6OOBNL6FGOKGJKM44OTBEISCRF5AKK4XESDPLJA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.834344079 (625h22m18.705427114s) [noise=true , address=45.91.116.9:4294 , piece_id=V6UOIFB3RQ5O7TM32BF7VIVBISWUGGKFTR7TEQ6NEGR7IWBJAZGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.845713768 (625h22m18.716796813s) [noise=true , address=45.91.116.14:4206 , piece_id=QZWX4P4CVXTCH3W3ZNBGKMPFIFNM7J2ETDEN3M7BNAQFNXBCEKIQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980140516 (625h22m18.851223551s) [noise=true , address=45.91.116.13:4161 , piece_id=E6KUOXKOUY5SPFEDZEFKTMYQKAA6WX7WQR7JRDOLHEYLHKQKCWBA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.980522623 (625h22m18.851605658s) [noise=true , address=145.79.227.30:4789 , piece_id=KDIW7L6BZBNYZTY6BQQUKM7RLFOXU5AAIGGDTV5ZKZCDKPCN5KFA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997884583 (625h22m18.868967618s) [noise=true , address=145.79.227.33:4875 , piece_id=WZIF5MDIWEHYGGOF3JYG4QB2NIOSJK4IAEDI3XS7NFCMOMOR74YQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: <nil> Dec 31 16:51:18.997904631 (625h22m18.868987666s) [seg_pos={0 0}, upload=547992]: Segment reached optimal threshold of 14 pieces. Dec 31 16:51:18.998092839 (625h22m18.869175874s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998099372 (625h22m18.869182417s) [noise=true , address=145.79.227.34:4916 , piece_id=HWX2NANK4KK2F7XJV2X3HMMZOSZMJFVLN3CFX6OOCPI47LNIEAGQ, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998204202 (625h22m18.869287237s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998210183 (625h22m18.869293218s) [noise=true , address=145.79.227.44:4199 , piece_id=CDG2EKVGRANZOMIBYZQNJA47P2JNB7FLHJTSSIDFIK5CPZG7AA2Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998213039 (625h22m18.869296074s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998221765 (625h22m18.86930481s) [noise=true , address=145.79.227.43:4163 , piece_id=GNOGMINWLF5ZXSPQBZS44ZM525HKJDCHCG6C5DXAXE7DBZWT36NA, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998312208 (625h22m18.869395243s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122S3PRcJwYfK5UEWESnYA7aTPHA6zCLFyBuFSm2Pb5wFSxDPiR): protocol: expected piece hash; context canceled; EOF Dec 31 16:51:18.998318149 (625h22m18.869401184s) [noise=true , address=5.231.73.3:4030 , piece_id=JOJGU35OWRYZRKLO7BQM5PIV5MIAK22XD3ILKHYKS5XJUUOTVI7Q, seg_pos={0 0}, upload=547992]: Long tail cancellation due to optimal threshold Dec 31 16:51:18.998356041 (625h22m18.869439076s) [seg_pos={0 0}, upload=547992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 16:51:18.998428418 (625h22m18.869511454s) [upload=547992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 16:51:18.998433228 (625h22m18.869516273s) [upload=547992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 16:51:19.285646870 (625h22m19.156729935s) [upload=547992]: Done uploading segments... Dec 31 16:51:19.285688400 (625h22m19.156771445s) [upload=547992]: Upload finished. err: <nil> Duration: 1m0.002962754s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 7.110370571s trace: https://jaeger.dev.storj.io/trace/5582e72abe7a8819 FAIL (2025-12-31 16:52:07 UTC) orbiter warning: context deadline exceeded Duration: 7.110400457s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.226168017s trace: https://jaeger.dev.storj.io/trace/5582e72abe7a8819 FAIL (2025-12-31 16:51:36 UTC) orbiter warning: context deadline exceeded Duration: 6.226197294s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)