Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.XX..................X...X..X..X..XX.X.X...XX.XX...X...X..] duration: 5.773401938s trace: https://jaeger.dev.storj.io/trace/51554ab69853858f FAIL (2025-12-30 08:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: S2GXHRS76MNFH46MFTN7RQMBY3Z2XR4YAIUMGPVI6SHVPLWKHODQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: SHI2ELFDXJIF4VWKIWVLOC32WET4FXY7J6SO42GZNIJFD37SFGXQ) context deadline exceeded ======LOGS====== ents... Dec 30 08:20:12.078958182 (592h51m11.950041227s) [upload=519745]: Got next segment. Inline: false Dec 30 08:20:12.079011604 (592h51m11.950094649s) [upload=519745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:20:12.079018828 (592h51m11.950101873s) [upload=519745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:20:12.321190003 (592h51m12.192273058s) [seg_pos={0 0}, upload=519745]: Begin upload segment... Dec 30 08:20:12.321262591 (592h51m12.192345626s) [seg_pos={0 0}, upload=519745]: Done begin upload segment. Dec 30 08:20:12.321326332 (592h51m12.192409377s) [noise=true , address=77.90.31.12:4590 , piece_id=I3SIKCGMBS3GZIDW67AE3WSFLJ47ZJVQA4LAXJ4XTVD7IWTI7QWQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.321347532 (592h51m12.192430577s) [noise=true , address=77.90.31.11:4553 , piece_id=LIQRORYDVNU2RS274W5GZIFFRNERGJKE6CSERVBFORKPLGT5AY2A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.322339782 (592h51m12.193422827s) [noise=true , address=45.91.116.10:4084 , piece_id=SMK7NH3UE6ZTE22RABYB3TIUP7OPIFUHC5525ACZXO5I3VGFRFMQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.322975883 (592h51m12.194058928s) [noise=true , address=77.90.31.17:4773 , piece_id=SHI2ELFDXJIF4VWKIWVLOC32WET4FXY7J6SO42GZNIJFD37SFGXQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.323610201 (592h51m12.194693246s) [noise=true , address=145.79.227.12:4765 , piece_id=XIR3A7YVMPRCW5ZELOPTDP3UX6PEIF4RDLOM7PCIGEQJILUFP5HA, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.324057072 (592h51m12.195140117s) [noise=true , address=145.79.227.11:4730 , piece_id=QWG62JWVYCA5HY3JBJDYOSFYDE3ARIVKWQ5TEZWMZP5KT7T4SV7Q, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.324507089 (592h51m12.195590134s) [noise=true , address=45.91.116.6:4975 , piece_id=AWTOE7BJJ63SV44VQUQBV5NPAHRXNXDZU37C4R7CXRWOZYEA5Y7A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.325033391 (592h51m12.196116436s) [noise=true , address=145.79.227.32:4836 , piece_id=YCK56ZSTSDC47HSL5EURIAVPIFELJ75GXKOQQSD4Q6TWYYGPKE2Q, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.326589083 (592h51m12.197672128s) [noise=true , address=145.79.227.36:4519 , piece_id=665O4ESBPCJGMXJA2MOGCWLIAQVH33CBIUDG5MZGA7WAOWCDOA5A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.326620253 (592h51m12.197703288s) [noise=true , address=45.91.116.13:4153 , piece_id=TC3L7VH7HWH3PBTME52QW4B3KZW3OVTCKYIJ447YJMFPSZFMBCQA, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.326701006 (592h51m12.197784041s) [noise=true , address=145.79.227.8:4331 , piece_id=R66EN6USQSK5O2LP5DG2UTLJ4C3TTHPIUVH6CQ3JWXRWHKHFMDNQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.328988512 (592h51m12.200071557s) [noise=true , address=77.90.31.15:4700 , piece_id=S2GXHRS76MNFH46MFTN7RQMBY3Z2XR4YAIUMGPVI6SHVPLWKHODQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.329145922 (592h51m12.200228967s) [noise=true , address=77.90.31.13:4627 , piece_id=O22XQXCUEA5FVSYF6ZJLBNADKI3IL36SBOJP7IDKEVMLV53SBXJQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.331522498 (592h51m12.202605543s) [noise=true , address=145.79.227.23:4545 , piece_id=R24UQN56HDHWXPEHNGAX52KLD2GJ5H3PTUMAWT6RZF72UVL4QOKQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.331840523 (592h51m12.202923568s) [noise=true , address=77.90.31.14:4661 , piece_id=M3GC7UK4LVWTVZX3UQGOZVO2ZPXPC7B4KTPSTPATYDWOM6BCLDDA, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.333613660 (592h51m12.204696715s) [noise=true , address=145.79.227.47:4266 , piece_id=2D76NHRDK7OE36MY4NJKTFDKJSS5A2GIPIN5AFQBXFKZTEMTDK7A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:14.034332705 (592h51m13.90541575s) [upload=519745]: Next returned nil segment Dec 30 08:20:14.034393371 (592h51m13.905476416s) [upload=519745]: Waiting for error group managing segments... Dec 30 08:20:14.316048940 (592h51m14.187131985s) [noise=true , address=145.79.227.8:4331 , piece_id=R66EN6USQSK5O2LP5DG2UTLJ4C3TTHPIUVH6CQ3JWXRWHKHFMDNQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.318434994 (592h51m14.189518039s) [noise=true , address=145.79.227.12:4765 , piece_id=XIR3A7YVMPRCW5ZELOPTDP3UX6PEIF4RDLOM7PCIGEQJILUFP5HA, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.318738111 (592h51m14.189821156s) [noise=true , address=145.79.227.23:4545 , piece_id=R24UQN56HDHWXPEHNGAX52KLD2GJ5H3PTUMAWT6RZF72UVL4QOKQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.325488214 (592h51m14.19657129s) [noise=true , address=145.79.227.11:4730 , piece_id=QWG62JWVYCA5HY3JBJDYOSFYDE3ARIVKWQ5TEZWMZP5KT7T4SV7Q, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.342170129 (592h51m14.213253175s) [noise=true , address=145.79.227.36:4519 , piece_id=665O4ESBPCJGMXJA2MOGCWLIAQVH33CBIUDG5MZGA7WAOWCDOA5A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.413693892 (592h51m14.284776937s) [noise=true , address=145.79.227.32:4836 , piece_id=YCK56ZSTSDC47HSL5EURIAVPIFELJ75GXKOQQSD4Q6TWYYGPKE2Q, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.428286990 (592h51m14.299370035s) [noise=true , address=145.79.227.47:4266 , piece_id=2D76NHRDK7OE36MY4NJKTFDKJSS5A2GIPIN5AFQBXFKZTEMTDK7A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.593533417 (592h51m14.464616462s) [noise=true , address=77.90.31.15:4700 , piece_id=S2GXHRS76MNFH46MFTN7RQMBY3Z2XR4YAIUMGPVI6SHVPLWKHODQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.603858568 (592h51m14.474941603s) [noise=true , address=77.90.31.14:4661 , piece_id=M3GC7UK4LVWTVZX3UQGOZVO2ZPXPC7B4KTPSTPATYDWOM6BCLDDA, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.628637341 (592h51m14.499720386s) [noise=true , address=77.90.31.12:4590 , piece_id=I3SIKCGMBS3GZIDW67AE3WSFLJ47ZJVQA4LAXJ4XTVD7IWTI7QWQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.632017968 (592h51m14.503101003s) [noise=true , address=77.90.31.13:4627 , piece_id=O22XQXCUEA5FVSYF6ZJLBNADKI3IL36SBOJP7IDKEVMLV53SBXJQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.737166088 (592h51m14.608249123s) [noise=true , address=45.91.116.6:4975 , piece_id=AWTOE7BJJ63SV44VQUQBV5NPAHRXNXDZU37C4R7CXRWOZYEA5Y7A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.969200420 (592h51m14.840283465s) [noise=true , address=77.90.31.17:4773 , piece_id=SHI2ELFDXJIF4VWKIWVLOC32WET4FXY7J6SO42GZNIJFD37SFGXQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.969223725 (592h51m14.84030677s) [seg_pos={0 0}, upload=519745]: Segment reached optimal threshold of 13 pieces. Dec 30 08:20:14.969484400 (592h51m14.840567446s) [noise=true , address=45.91.116.10:4084 , piece_id=SMK7NH3UE6ZTE22RABYB3TIUP7OPIFUHC5525ACZXO5I3VGFRFMQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:57048->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:57048->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57048->45.91.116.10:4084: use of closed network connection Dec 30 08:20:14.969493769 (592h51m14.840576814s) [noise=true , address=45.91.116.10:4084 , piece_id=SMK7NH3UE6ZTE22RABYB3TIUP7OPIFUHC5525ACZXO5I3VGFRFMQ, seg_pos={0 0}, upload=519745]: Long tail cancellation due to optimal threshold Dec 30 08:20:14.969569784 (592h51m14.840652819s) [noise=true , address=45.91.116.13:4153 , piece_id=TC3L7VH7HWH3PBTME52QW4B3KZW3OVTCKYIJ447YJMFPSZFMBCQA, seg_pos={0 0}, upload=519745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:38024->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:38024->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38024->45.91.116.13:4153: use of closed network connection Dec 30 08:20:14.969601724 (592h51m14.84068479s) [noise=true , address=45.91.116.13:4153 , piece_id=TC3L7VH7HWH3PBTME52QW4B3KZW3OVTCKYIJ447YJMFPSZFMBCQA, seg_pos={0 0}, upload=519745]: Long tail cancellation due to optimal threshold Dec 30 08:20:14.969667419 (592h51m14.840750464s) [noise=true , address=77.90.31.11:4553 , piece_id=LIQRORYDVNU2RS274W5GZIFFRNERGJKE6CSERVBFORKPLGT5AY2A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 08:20:14.969677529 (592h51m14.840760574s) [noise=true , address=77.90.31.11:4553 , piece_id=LIQRORYDVNU2RS274W5GZIFFRNERGJKE6CSERVBFORKPLGT5AY2A, seg_pos={0 0}, upload=519745]: Long tail cancellation due to optimal threshold Dec 30 08:20:14.969773562 (592h51m14.840856597s) [seg_pos={0 0}, upload=519745]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:20:14.969881357 (592h51m14.840964392s) [upload=519745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:20:14.969886216 (592h51m14.840969261s) [upload=519745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:20:15.252074139 (592h51m15.123157204s) [upload=519745]: Done uploading segments... Dec 30 08:20:15.252099458 (592h51m15.123182503s) [upload=519745]: Upload finished. err: <nil> Duration: 5.773422998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X................X..................] duration: 34.410845092s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:21:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 34.410873186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.XXX.XXX.X.XXXXXX.XX.......X..X..X..................X.......] duration: 3.520273533s trace: https://jaeger.dev.storj.io/trace/7a032e32a5763d1a FAIL (2025-12-30 08:21:04 UTC) orbiter warning: context deadline exceeded Duration: 3.520295725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX.XX..................X...X..X..X..XX.X.X...XX.XX...X...X.] duration: 5.905255764s trace: https://jaeger.dev.storj.io/trace/51554ab69853858f FAIL (2025-12-30 08:20:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: S2GXHRS76MNFH46MFTN7RQMBY3Z2XR4YAIUMGPVI6SHVPLWKHODQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: SHI2ELFDXJIF4VWKIWVLOC32WET4FXY7J6SO42GZNIJFD37SFGXQ) context deadline exceeded ======LOGS====== ents... Dec 30 08:20:12.078958182 (592h51m11.950041227s) [upload=519745]: Got next segment. Inline: false Dec 30 08:20:12.079011604 (592h51m11.950094649s) [upload=519745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:20:12.079018828 (592h51m11.950101873s) [upload=519745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:20:12.321190003 (592h51m12.192273058s) [seg_pos={0 0}, upload=519745]: Begin upload segment... Dec 30 08:20:12.321262591 (592h51m12.192345626s) [seg_pos={0 0}, upload=519745]: Done begin upload segment. Dec 30 08:20:12.321326332 (592h51m12.192409377s) [noise=true , address=77.90.31.12:4590 , piece_id=I3SIKCGMBS3GZIDW67AE3WSFLJ47ZJVQA4LAXJ4XTVD7IWTI7QWQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.321347532 (592h51m12.192430577s) [noise=true , address=77.90.31.11:4553 , piece_id=LIQRORYDVNU2RS274W5GZIFFRNERGJKE6CSERVBFORKPLGT5AY2A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.322339782 (592h51m12.193422827s) [noise=true , address=45.91.116.10:4084 , piece_id=SMK7NH3UE6ZTE22RABYB3TIUP7OPIFUHC5525ACZXO5I3VGFRFMQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.322975883 (592h51m12.194058928s) [noise=true , address=77.90.31.17:4773 , piece_id=SHI2ELFDXJIF4VWKIWVLOC32WET4FXY7J6SO42GZNIJFD37SFGXQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.323610201 (592h51m12.194693246s) [noise=true , address=145.79.227.12:4765 , piece_id=XIR3A7YVMPRCW5ZELOPTDP3UX6PEIF4RDLOM7PCIGEQJILUFP5HA, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.324057072 (592h51m12.195140117s) [noise=true , address=145.79.227.11:4730 , piece_id=QWG62JWVYCA5HY3JBJDYOSFYDE3ARIVKWQ5TEZWMZP5KT7T4SV7Q, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.324507089 (592h51m12.195590134s) [noise=true , address=45.91.116.6:4975 , piece_id=AWTOE7BJJ63SV44VQUQBV5NPAHRXNXDZU37C4R7CXRWOZYEA5Y7A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.325033391 (592h51m12.196116436s) [noise=true , address=145.79.227.32:4836 , piece_id=YCK56ZSTSDC47HSL5EURIAVPIFELJ75GXKOQQSD4Q6TWYYGPKE2Q, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.326589083 (592h51m12.197672128s) [noise=true , address=145.79.227.36:4519 , piece_id=665O4ESBPCJGMXJA2MOGCWLIAQVH33CBIUDG5MZGA7WAOWCDOA5A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.326620253 (592h51m12.197703288s) [noise=true , address=45.91.116.13:4153 , piece_id=TC3L7VH7HWH3PBTME52QW4B3KZW3OVTCKYIJ447YJMFPSZFMBCQA, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.326701006 (592h51m12.197784041s) [noise=true , address=145.79.227.8:4331 , piece_id=R66EN6USQSK5O2LP5DG2UTLJ4C3TTHPIUVH6CQ3JWXRWHKHFMDNQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.328988512 (592h51m12.200071557s) [noise=true , address=77.90.31.15:4700 , piece_id=S2GXHRS76MNFH46MFTN7RQMBY3Z2XR4YAIUMGPVI6SHVPLWKHODQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.329145922 (592h51m12.200228967s) [noise=true , address=77.90.31.13:4627 , piece_id=O22XQXCUEA5FVSYF6ZJLBNADKI3IL36SBOJP7IDKEVMLV53SBXJQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.331522498 (592h51m12.202605543s) [noise=true , address=145.79.227.23:4545 , piece_id=R24UQN56HDHWXPEHNGAX52KLD2GJ5H3PTUMAWT6RZF72UVL4QOKQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.331840523 (592h51m12.202923568s) [noise=true , address=77.90.31.14:4661 , piece_id=M3GC7UK4LVWTVZX3UQGOZVO2ZPXPC7B4KTPSTPATYDWOM6BCLDDA, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.333613660 (592h51m12.204696715s) [noise=true , address=145.79.227.47:4266 , piece_id=2D76NHRDK7OE36MY4NJKTFDKJSS5A2GIPIN5AFQBXFKZTEMTDK7A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:14.034332705 (592h51m13.90541575s) [upload=519745]: Next returned nil segment Dec 30 08:20:14.034393371 (592h51m13.905476416s) [upload=519745]: Waiting for error group managing segments... Dec 30 08:20:14.316048940 (592h51m14.187131985s) [noise=true , address=145.79.227.8:4331 , piece_id=R66EN6USQSK5O2LP5DG2UTLJ4C3TTHPIUVH6CQ3JWXRWHKHFMDNQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.318434994 (592h51m14.189518039s) [noise=true , address=145.79.227.12:4765 , piece_id=XIR3A7YVMPRCW5ZELOPTDP3UX6PEIF4RDLOM7PCIGEQJILUFP5HA, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.318738111 (592h51m14.189821156s) [noise=true , address=145.79.227.23:4545 , piece_id=R24UQN56HDHWXPEHNGAX52KLD2GJ5H3PTUMAWT6RZF72UVL4QOKQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.325488214 (592h51m14.19657129s) [noise=true , address=145.79.227.11:4730 , piece_id=QWG62JWVYCA5HY3JBJDYOSFYDE3ARIVKWQ5TEZWMZP5KT7T4SV7Q, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.342170129 (592h51m14.213253175s) [noise=true , address=145.79.227.36:4519 , piece_id=665O4ESBPCJGMXJA2MOGCWLIAQVH33CBIUDG5MZGA7WAOWCDOA5A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.413693892 (592h51m14.284776937s) [noise=true , address=145.79.227.32:4836 , piece_id=YCK56ZSTSDC47HSL5EURIAVPIFELJ75GXKOQQSD4Q6TWYYGPKE2Q, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.428286990 (592h51m14.299370035s) [noise=true , address=145.79.227.47:4266 , piece_id=2D76NHRDK7OE36MY4NJKTFDKJSS5A2GIPIN5AFQBXFKZTEMTDK7A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.593533417 (592h51m14.464616462s) [noise=true , address=77.90.31.15:4700 , piece_id=S2GXHRS76MNFH46MFTN7RQMBY3Z2XR4YAIUMGPVI6SHVPLWKHODQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.603858568 (592h51m14.474941603s) [noise=true , address=77.90.31.14:4661 , piece_id=M3GC7UK4LVWTVZX3UQGOZVO2ZPXPC7B4KTPSTPATYDWOM6BCLDDA, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.628637341 (592h51m14.499720386s) [noise=true , address=77.90.31.12:4590 , piece_id=I3SIKCGMBS3GZIDW67AE3WSFLJ47ZJVQA4LAXJ4XTVD7IWTI7QWQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.632017968 (592h51m14.503101003s) [noise=true , address=77.90.31.13:4627 , piece_id=O22XQXCUEA5FVSYF6ZJLBNADKI3IL36SBOJP7IDKEVMLV53SBXJQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.737166088 (592h51m14.608249123s) [noise=true , address=45.91.116.6:4975 , piece_id=AWTOE7BJJ63SV44VQUQBV5NPAHRXNXDZU37C4R7CXRWOZYEA5Y7A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.969200420 (592h51m14.840283465s) [noise=true , address=77.90.31.17:4773 , piece_id=SHI2ELFDXJIF4VWKIWVLOC32WET4FXY7J6SO42GZNIJFD37SFGXQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.969223725 (592h51m14.84030677s) [seg_pos={0 0}, upload=519745]: Segment reached optimal threshold of 13 pieces. Dec 30 08:20:14.969484400 (592h51m14.840567446s) [noise=true , address=45.91.116.10:4084 , piece_id=SMK7NH3UE6ZTE22RABYB3TIUP7OPIFUHC5525ACZXO5I3VGFRFMQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:57048->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:57048->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57048->45.91.116.10:4084: use of closed network connection Dec 30 08:20:14.969493769 (592h51m14.840576814s) [noise=true , address=45.91.116.10:4084 , piece_id=SMK7NH3UE6ZTE22RABYB3TIUP7OPIFUHC5525ACZXO5I3VGFRFMQ, seg_pos={0 0}, upload=519745]: Long tail cancellation due to optimal threshold Dec 30 08:20:14.969569784 (592h51m14.840652819s) [noise=true , address=45.91.116.13:4153 , piece_id=TC3L7VH7HWH3PBTME52QW4B3KZW3OVTCKYIJ447YJMFPSZFMBCQA, seg_pos={0 0}, upload=519745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:38024->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:38024->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38024->45.91.116.13:4153: use of closed network connection Dec 30 08:20:14.969601724 (592h51m14.84068479s) [noise=true , address=45.91.116.13:4153 , piece_id=TC3L7VH7HWH3PBTME52QW4B3KZW3OVTCKYIJ447YJMFPSZFMBCQA, seg_pos={0 0}, upload=519745]: Long tail cancellation due to optimal threshold Dec 30 08:20:14.969667419 (592h51m14.840750464s) [noise=true , address=77.90.31.11:4553 , piece_id=LIQRORYDVNU2RS274W5GZIFFRNERGJKE6CSERVBFORKPLGT5AY2A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 08:20:14.969677529 (592h51m14.840760574s) [noise=true , address=77.90.31.11:4553 , piece_id=LIQRORYDVNU2RS274W5GZIFFRNERGJKE6CSERVBFORKPLGT5AY2A, seg_pos={0 0}, upload=519745]: Long tail cancellation due to optimal threshold Dec 30 08:20:14.969773562 (592h51m14.840856597s) [seg_pos={0 0}, upload=519745]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:20:14.969881357 (592h51m14.840964392s) [upload=519745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:20:14.969886216 (592h51m14.840969261s) [upload=519745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:20:15.252074139 (592h51m15.123157204s) [upload=519745]: Done uploading segments... Dec 30 08:20:15.252099458 (592h51m15.123182503s) [upload=519745]: Upload finished. err: <nil> Duration: 5.905283276s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X.XXX.XXX.X.XXXXXX.XX.......X..X..X..................X......] duration: 3.426194542s trace: https://jaeger.dev.storj.io/trace/7a032e32a5763d1a FAIL (2025-12-30 08:20:44 UTC) orbiter warning: context deadline exceeded Duration: 3.426295434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX.XX..................X...X..X..X..XX.X.X...XX.XX...X...X] duration: 30.00350712s trace: https://jaeger.dev.storj.io/trace/51554ab69853858f FAIL (2025-12-30 08:20:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: S2GXHRS76MNFH46MFTN7RQMBY3Z2XR4YAIUMGPVI6SHVPLWKHODQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: SHI2ELFDXJIF4VWKIWVLOC32WET4FXY7J6SO42GZNIJFD37SFGXQ) context deadline exceeded ======LOGS====== ents... Dec 30 08:20:12.078958182 (592h51m11.950041227s) [upload=519745]: Got next segment. Inline: false Dec 30 08:20:12.079011604 (592h51m11.950094649s) [upload=519745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:20:12.079018828 (592h51m11.950101873s) [upload=519745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:20:12.321190003 (592h51m12.192273058s) [seg_pos={0 0}, upload=519745]: Begin upload segment... Dec 30 08:20:12.321262591 (592h51m12.192345626s) [seg_pos={0 0}, upload=519745]: Done begin upload segment. Dec 30 08:20:12.321326332 (592h51m12.192409377s) [noise=true , address=77.90.31.12:4590 , piece_id=I3SIKCGMBS3GZIDW67AE3WSFLJ47ZJVQA4LAXJ4XTVD7IWTI7QWQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.321347532 (592h51m12.192430577s) [noise=true , address=77.90.31.11:4553 , piece_id=LIQRORYDVNU2RS274W5GZIFFRNERGJKE6CSERVBFORKPLGT5AY2A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.322339782 (592h51m12.193422827s) [noise=true , address=45.91.116.10:4084 , piece_id=SMK7NH3UE6ZTE22RABYB3TIUP7OPIFUHC5525ACZXO5I3VGFRFMQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.322975883 (592h51m12.194058928s) [noise=true , address=77.90.31.17:4773 , piece_id=SHI2ELFDXJIF4VWKIWVLOC32WET4FXY7J6SO42GZNIJFD37SFGXQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.323610201 (592h51m12.194693246s) [noise=true , address=145.79.227.12:4765 , piece_id=XIR3A7YVMPRCW5ZELOPTDP3UX6PEIF4RDLOM7PCIGEQJILUFP5HA, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.324057072 (592h51m12.195140117s) [noise=true , address=145.79.227.11:4730 , piece_id=QWG62JWVYCA5HY3JBJDYOSFYDE3ARIVKWQ5TEZWMZP5KT7T4SV7Q, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.324507089 (592h51m12.195590134s) [noise=true , address=45.91.116.6:4975 , piece_id=AWTOE7BJJ63SV44VQUQBV5NPAHRXNXDZU37C4R7CXRWOZYEA5Y7A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.325033391 (592h51m12.196116436s) [noise=true , address=145.79.227.32:4836 , piece_id=YCK56ZSTSDC47HSL5EURIAVPIFELJ75GXKOQQSD4Q6TWYYGPKE2Q, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.326589083 (592h51m12.197672128s) [noise=true , address=145.79.227.36:4519 , piece_id=665O4ESBPCJGMXJA2MOGCWLIAQVH33CBIUDG5MZGA7WAOWCDOA5A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.326620253 (592h51m12.197703288s) [noise=true , address=45.91.116.13:4153 , piece_id=TC3L7VH7HWH3PBTME52QW4B3KZW3OVTCKYIJ447YJMFPSZFMBCQA, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.326701006 (592h51m12.197784041s) [noise=true , address=145.79.227.8:4331 , piece_id=R66EN6USQSK5O2LP5DG2UTLJ4C3TTHPIUVH6CQ3JWXRWHKHFMDNQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.328988512 (592h51m12.200071557s) [noise=true , address=77.90.31.15:4700 , piece_id=S2GXHRS76MNFH46MFTN7RQMBY3Z2XR4YAIUMGPVI6SHVPLWKHODQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.329145922 (592h51m12.200228967s) [noise=true , address=77.90.31.13:4627 , piece_id=O22XQXCUEA5FVSYF6ZJLBNADKI3IL36SBOJP7IDKEVMLV53SBXJQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.331522498 (592h51m12.202605543s) [noise=true , address=145.79.227.23:4545 , piece_id=R24UQN56HDHWXPEHNGAX52KLD2GJ5H3PTUMAWT6RZF72UVL4QOKQ, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.331840523 (592h51m12.202923568s) [noise=true , address=77.90.31.14:4661 , piece_id=M3GC7UK4LVWTVZX3UQGOZVO2ZPXPC7B4KTPSTPATYDWOM6BCLDDA, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:12.333613660 (592h51m12.204696715s) [noise=true , address=145.79.227.47:4266 , piece_id=2D76NHRDK7OE36MY4NJKTFDKJSS5A2GIPIN5AFQBXFKZTEMTDK7A, seg_pos={0 0}, upload=519745]: Uploading piece... Dec 30 08:20:14.034332705 (592h51m13.90541575s) [upload=519745]: Next returned nil segment Dec 30 08:20:14.034393371 (592h51m13.905476416s) [upload=519745]: Waiting for error group managing segments... Dec 30 08:20:14.316048940 (592h51m14.187131985s) [noise=true , address=145.79.227.8:4331 , piece_id=R66EN6USQSK5O2LP5DG2UTLJ4C3TTHPIUVH6CQ3JWXRWHKHFMDNQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.318434994 (592h51m14.189518039s) [noise=true , address=145.79.227.12:4765 , piece_id=XIR3A7YVMPRCW5ZELOPTDP3UX6PEIF4RDLOM7PCIGEQJILUFP5HA, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.318738111 (592h51m14.189821156s) [noise=true , address=145.79.227.23:4545 , piece_id=R24UQN56HDHWXPEHNGAX52KLD2GJ5H3PTUMAWT6RZF72UVL4QOKQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.325488214 (592h51m14.19657129s) [noise=true , address=145.79.227.11:4730 , piece_id=QWG62JWVYCA5HY3JBJDYOSFYDE3ARIVKWQ5TEZWMZP5KT7T4SV7Q, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.342170129 (592h51m14.213253175s) [noise=true , address=145.79.227.36:4519 , piece_id=665O4ESBPCJGMXJA2MOGCWLIAQVH33CBIUDG5MZGA7WAOWCDOA5A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.413693892 (592h51m14.284776937s) [noise=true , address=145.79.227.32:4836 , piece_id=YCK56ZSTSDC47HSL5EURIAVPIFELJ75GXKOQQSD4Q6TWYYGPKE2Q, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.428286990 (592h51m14.299370035s) [noise=true , address=145.79.227.47:4266 , piece_id=2D76NHRDK7OE36MY4NJKTFDKJSS5A2GIPIN5AFQBXFKZTEMTDK7A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.593533417 (592h51m14.464616462s) [noise=true , address=77.90.31.15:4700 , piece_id=S2GXHRS76MNFH46MFTN7RQMBY3Z2XR4YAIUMGPVI6SHVPLWKHODQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.603858568 (592h51m14.474941603s) [noise=true , address=77.90.31.14:4661 , piece_id=M3GC7UK4LVWTVZX3UQGOZVO2ZPXPC7B4KTPSTPATYDWOM6BCLDDA, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.628637341 (592h51m14.499720386s) [noise=true , address=77.90.31.12:4590 , piece_id=I3SIKCGMBS3GZIDW67AE3WSFLJ47ZJVQA4LAXJ4XTVD7IWTI7QWQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.632017968 (592h51m14.503101003s) [noise=true , address=77.90.31.13:4627 , piece_id=O22XQXCUEA5FVSYF6ZJLBNADKI3IL36SBOJP7IDKEVMLV53SBXJQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.737166088 (592h51m14.608249123s) [noise=true , address=45.91.116.6:4975 , piece_id=AWTOE7BJJ63SV44VQUQBV5NPAHRXNXDZU37C4R7CXRWOZYEA5Y7A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.969200420 (592h51m14.840283465s) [noise=true , address=77.90.31.17:4773 , piece_id=SHI2ELFDXJIF4VWKIWVLOC32WET4FXY7J6SO42GZNIJFD37SFGXQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: <nil> Dec 30 08:20:14.969223725 (592h51m14.84030677s) [seg_pos={0 0}, upload=519745]: Segment reached optimal threshold of 13 pieces. Dec 30 08:20:14.969484400 (592h51m14.840567446s) [noise=true , address=45.91.116.10:4084 , piece_id=SMK7NH3UE6ZTE22RABYB3TIUP7OPIFUHC5525ACZXO5I3VGFRFMQ, seg_pos={0 0}, upload=519745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:57048->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:57048->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57048->45.91.116.10:4084: use of closed network connection Dec 30 08:20:14.969493769 (592h51m14.840576814s) [noise=true , address=45.91.116.10:4084 , piece_id=SMK7NH3UE6ZTE22RABYB3TIUP7OPIFUHC5525ACZXO5I3VGFRFMQ, seg_pos={0 0}, upload=519745]: Long tail cancellation due to optimal threshold Dec 30 08:20:14.969569784 (592h51m14.840652819s) [noise=true , address=45.91.116.13:4153 , piece_id=TC3L7VH7HWH3PBTME52QW4B3KZW3OVTCKYIJ447YJMFPSZFMBCQA, seg_pos={0 0}, upload=519745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:38024->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:38024->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38024->45.91.116.13:4153: use of closed network connection Dec 30 08:20:14.969601724 (592h51m14.84068479s) [noise=true , address=45.91.116.13:4153 , piece_id=TC3L7VH7HWH3PBTME52QW4B3KZW3OVTCKYIJ447YJMFPSZFMBCQA, seg_pos={0 0}, upload=519745]: Long tail cancellation due to optimal threshold Dec 30 08:20:14.969667419 (592h51m14.840750464s) [noise=true , address=77.90.31.11:4553 , piece_id=LIQRORYDVNU2RS274W5GZIFFRNERGJKE6CSERVBFORKPLGT5AY2A, seg_pos={0 0}, upload=519745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 08:20:14.969677529 (592h51m14.840760574s) [noise=true , address=77.90.31.11:4553 , piece_id=LIQRORYDVNU2RS274W5GZIFFRNERGJKE6CSERVBFORKPLGT5AY2A, seg_pos={0 0}, upload=519745]: Long tail cancellation due to optimal threshold Dec 30 08:20:14.969773562 (592h51m14.840856597s) [seg_pos={0 0}, upload=519745]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:20:14.969881357 (592h51m14.840964392s) [upload=519745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:20:14.969886216 (592h51m14.840969261s) [upload=519745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:20:15.252074139 (592h51m15.123157204s) [upload=519745]: Done uploading segments... Dec 30 08:20:15.252099458 (592h51m15.123182503s) [upload=519745]: Upload finished. err: <nil> Duration: 30.003533539s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X................X.................] duration: 40.982611096s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:20:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 40.982644269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [XX.XXX.XXX.X.XXXXXX.XX.......X..X..X..................X.....] duration: 3.472369787s trace: https://jaeger.dev.storj.io/trace/7a032e32a5763d1a FAIL (2025-12-30 08:20:19 UTC) orbiter warning: context deadline exceeded Duration: 3.472399042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...XX.XX..................X...X..X..X..XX.X.X...XX.XX...X...] duration: 5.990914509s trace: https://jaeger.dev.storj.io/trace/20249d585c5f6db FAIL (2025-12-30 08:19:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA) context deadline exceeded ======LOGS====== Dec 30 08:17:00.846789529 (592h48m0.717872574s) [upload=519699]: Starting upload Dec 30 08:17:00.846881204 (592h48m0.717964249s) [upload=519699]: Done starting upload Dec 30 08:17:00.846963620 (592h48m0.718046665s) [upload=519699]: Uploading segments... Dec 30 08:17:00.847283640 (592h48m0.718366685s) [upload=519699]: Got next segment. Inline: false Dec 30 08:17:00.847371628 (592h48m0.718454673s) [upload=519699]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:17:00.847377619 (592h48m0.718460664s) [upload=519699]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:17:01.091255865 (592h48m0.96233891s) [seg_pos={0 0}, upload=519699]: Begin upload segment... Dec 30 08:17:01.091316200 (592h48m0.962399235s) [seg_pos={0 0}, upload=519699]: Done begin upload segment. Dec 30 08:17:01.091353702 (592h48m0.962436767s) [noise=true , address=145.79.227.41:4086 , piece_id=TF63E7J6W7R65MMLD47IXWSUP3DPKS2AOJ3VDIGMYN2HBKBFJGUQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.091424436 (592h48m0.962507481s) [noise=true , address=45.91.116.11:4119 , piece_id=NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.092107618 (592h48m0.963190663s) [noise=true , address=145.79.227.13:4802 , piece_id=DTYJK3TJBATMICP7CIKYJUMOGTKJDAOBENZTXN2RQASNH7E5DXSA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.092584986 (592h48m0.963668031s) [noise=true , address=145.79.227.23:4547 , piece_id=W34NASX5NCYNXTDS3V7QNYE3CA7XB5WVGXWPYPVWF5JJ4NOZIIDA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093019474 (592h48m0.964102519s) [noise=true , address=45.91.116.14:4192 , piece_id=POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093486252 (592h48m0.964569287s) [noise=true , address=145.79.227.3:4150 , piece_id=G7UZ2RHTY2BBFTTJRRCFKIO7FCUEJJWCERLU3BPFM2OMYQ64CD4A, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093976446 (592h48m0.965059491s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.094485345 (592h48m0.96556838s) [noise=true , address=145.79.227.14:4838 , piece_id=BTGTBI6Y53DFPDMFJTFV2AOECHL3YSHNAH2AV4AKSXXOVYOBFZHA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.095153427 (592h48m0.966236482s) [noise=true , address=145.79.227.25:4617 , piece_id=SEQXHIVLB2IFFS7AFBTRPT2NG7AQMYF4G4Q6X3ZDAC253J45GVMA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.095714736 (592h48m0.966797781s) [noise=true , address=145.79.227.39:4013 , piece_id=PPIXK4GRVLCBMBWKXESUK6OOV75QOPY3N7CDMKYEXJAAZXIZMKCA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.096269773 (592h48m0.967352818s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.096830851 (592h48m0.967913886s) [noise=true , address=145.79.227.10:4401 , piece_id=INBNWHF5KY5VKN7KRGAWA73MBU7JW7S22C4DK7BFAAQ3VTZIDGJQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.097331084 (592h48m0.968414129s) [noise=true , address=45.91.116.5:4298 , piece_id=GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.098069299 (592h48m0.969152375s) [noise=true , address=77.90.31.14:4662 , piece_id=R7URHPL2SYNQOUVRIF3DNKYIQW2AXKDSXROHJSBBNQKVN2IX7YAA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.098662018 (592h48m0.969745053s) [noise=true , address=145.79.227.52:4447 , piece_id=YHBKCKPPWKZIDUI2RSZ64Y7TEWPU23L7CFFX34KR63T3AU7O4FRA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.100199015 (592h48m0.97128205s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:02.800462773 (592h48m2.671545818s) [upload=519699]: Next returned nil segment Dec 30 08:17:02.800482280 (592h48m2.671565325s) [upload=519699]: Waiting for error group managing segments... Dec 30 08:17:03.079896141 (592h48m2.950979197s) [noise=true , address=145.79.227.3:4150 , piece_id=G7UZ2RHTY2BBFTTJRRCFKIO7FCUEJJWCERLU3BPFM2OMYQ64CD4A, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.091948050 (592h48m2.963031105s) [noise=true , address=145.79.227.10:4401 , piece_id=INBNWHF5KY5VKN7KRGAWA73MBU7JW7S22C4DK7BFAAQ3VTZIDGJQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.095491658 (592h48m2.966574703s) [noise=true , address=145.79.227.25:4617 , piece_id=SEQXHIVLB2IFFS7AFBTRPT2NG7AQMYF4G4Q6X3ZDAC253J45GVMA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.104424257 (592h48m2.975507302s) [noise=true , address=145.79.227.39:4013 , piece_id=PPIXK4GRVLCBMBWKXESUK6OOV75QOPY3N7CDMKYEXJAAZXIZMKCA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.163414565 (592h48m3.03449761s) [noise=true , address=145.79.227.13:4802 , piece_id=DTYJK3TJBATMICP7CIKYJUMOGTKJDAOBENZTXN2RQASNH7E5DXSA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.165365981 (592h48m3.036449016s) [noise=true , address=145.79.227.41:4086 , piece_id=TF63E7J6W7R65MMLD47IXWSUP3DPKS2AOJ3VDIGMYN2HBKBFJGUQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.177755521 (592h48m3.048838566s) [noise=true , address=145.79.227.14:4838 , piece_id=BTGTBI6Y53DFPDMFJTFV2AOECHL3YSHNAH2AV4AKSXXOVYOBFZHA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.178258179 (592h48m3.049341224s) [noise=true , address=145.79.227.23:4547 , piece_id=W34NASX5NCYNXTDS3V7QNYE3CA7XB5WVGXWPYPVWF5JJ4NOZIIDA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.195296913 (592h48m3.066379958s) [noise=true , address=145.79.227.52:4447 , piece_id=YHBKCKPPWKZIDUI2RSZ64Y7TEWPU23L7CFFX34KR63T3AU7O4FRA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.300032047 (592h48m3.171115093s) [noise=true , address=77.90.31.14:4662 , piece_id=R7URHPL2SYNQOUVRIF3DNKYIQW2AXKDSXROHJSBBNQKVN2IX7YAA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.432987406 (592h48m3.304070451s) [noise=true , address=45.91.116.5:4298 , piece_id=GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.524295082 (592h48m3.395378127s) [noise=true , address=45.91.116.14:4192 , piece_id=POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.532208229 (592h48m3.403291274s) [noise=true , address=45.91.116.11:4119 , piece_id=NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.532228268 (592h48m3.403311303s) [seg_pos={0 0}, upload=519699]: Segment reached optimal threshold of 13 pieces. Dec 30 08:17:03.532433178 (592h48m3.403516223s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 08:17:03.532443237 (592h48m3.403526272s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532597912 (592h48m3.403680947s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 08:17:03.532605556 (592h48m3.403688592s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532899436 (592h48m3.403982481s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection Dec 30 08:17:03.532919434 (592h48m3.404002479s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532977074 (592h48m3.404060109s) [seg_pos={0 0}, upload=519699]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:17:03.533149371 (592h48m3.404232416s) [upload=519699]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:17:03.533157207 (592h48m3.404240242s) [upload=519699]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:17:03.817060166 (592h48m3.688143211s) [upload=519699]: Done uploading segments... Dec 30 08:17:03.817089281 (592h48m3.688172326s) [upload=519699]: Upload finished. err: <nil> Duration: 5.990952441s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X..................X....] duration: 3.557931219s trace: https://jaeger.dev.storj.io/trace/7a032e32a5763d1a FAIL (2025-12-30 08:19:20 UTC) orbiter warning: context deadline exceeded Duration: 3.557955354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X................X................] duration: 36.246206964s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:19:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 36.246233606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....XX.XX..................X...X..X..X..XX.X.X...XX.XX...X..] duration: 5.979174994s trace: https://jaeger.dev.storj.io/trace/20249d585c5f6db FAIL (2025-12-30 08:18:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA) context deadline exceeded ======LOGS====== Dec 30 08:17:00.846789529 (592h48m0.717872574s) [upload=519699]: Starting upload Dec 30 08:17:00.846881204 (592h48m0.717964249s) [upload=519699]: Done starting upload Dec 30 08:17:00.846963620 (592h48m0.718046665s) [upload=519699]: Uploading segments... Dec 30 08:17:00.847283640 (592h48m0.718366685s) [upload=519699]: Got next segment. Inline: false Dec 30 08:17:00.847371628 (592h48m0.718454673s) [upload=519699]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:17:00.847377619 (592h48m0.718460664s) [upload=519699]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:17:01.091255865 (592h48m0.96233891s) [seg_pos={0 0}, upload=519699]: Begin upload segment... Dec 30 08:17:01.091316200 (592h48m0.962399235s) [seg_pos={0 0}, upload=519699]: Done begin upload segment. Dec 30 08:17:01.091353702 (592h48m0.962436767s) [noise=true , address=145.79.227.41:4086 , piece_id=TF63E7J6W7R65MMLD47IXWSUP3DPKS2AOJ3VDIGMYN2HBKBFJGUQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.091424436 (592h48m0.962507481s) [noise=true , address=45.91.116.11:4119 , piece_id=NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.092107618 (592h48m0.963190663s) [noise=true , address=145.79.227.13:4802 , piece_id=DTYJK3TJBATMICP7CIKYJUMOGTKJDAOBENZTXN2RQASNH7E5DXSA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.092584986 (592h48m0.963668031s) [noise=true , address=145.79.227.23:4547 , piece_id=W34NASX5NCYNXTDS3V7QNYE3CA7XB5WVGXWPYPVWF5JJ4NOZIIDA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093019474 (592h48m0.964102519s) [noise=true , address=45.91.116.14:4192 , piece_id=POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093486252 (592h48m0.964569287s) [noise=true , address=145.79.227.3:4150 , piece_id=G7UZ2RHTY2BBFTTJRRCFKIO7FCUEJJWCERLU3BPFM2OMYQ64CD4A, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093976446 (592h48m0.965059491s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.094485345 (592h48m0.96556838s) [noise=true , address=145.79.227.14:4838 , piece_id=BTGTBI6Y53DFPDMFJTFV2AOECHL3YSHNAH2AV4AKSXXOVYOBFZHA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.095153427 (592h48m0.966236482s) [noise=true , address=145.79.227.25:4617 , piece_id=SEQXHIVLB2IFFS7AFBTRPT2NG7AQMYF4G4Q6X3ZDAC253J45GVMA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.095714736 (592h48m0.966797781s) [noise=true , address=145.79.227.39:4013 , piece_id=PPIXK4GRVLCBMBWKXESUK6OOV75QOPY3N7CDMKYEXJAAZXIZMKCA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.096269773 (592h48m0.967352818s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.096830851 (592h48m0.967913886s) [noise=true , address=145.79.227.10:4401 , piece_id=INBNWHF5KY5VKN7KRGAWA73MBU7JW7S22C4DK7BFAAQ3VTZIDGJQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.097331084 (592h48m0.968414129s) [noise=true , address=45.91.116.5:4298 , piece_id=GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.098069299 (592h48m0.969152375s) [noise=true , address=77.90.31.14:4662 , piece_id=R7URHPL2SYNQOUVRIF3DNKYIQW2AXKDSXROHJSBBNQKVN2IX7YAA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.098662018 (592h48m0.969745053s) [noise=true , address=145.79.227.52:4447 , piece_id=YHBKCKPPWKZIDUI2RSZ64Y7TEWPU23L7CFFX34KR63T3AU7O4FRA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.100199015 (592h48m0.97128205s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:02.800462773 (592h48m2.671545818s) [upload=519699]: Next returned nil segment Dec 30 08:17:02.800482280 (592h48m2.671565325s) [upload=519699]: Waiting for error group managing segments... Dec 30 08:17:03.079896141 (592h48m2.950979197s) [noise=true , address=145.79.227.3:4150 , piece_id=G7UZ2RHTY2BBFTTJRRCFKIO7FCUEJJWCERLU3BPFM2OMYQ64CD4A, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.091948050 (592h48m2.963031105s) [noise=true , address=145.79.227.10:4401 , piece_id=INBNWHF5KY5VKN7KRGAWA73MBU7JW7S22C4DK7BFAAQ3VTZIDGJQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.095491658 (592h48m2.966574703s) [noise=true , address=145.79.227.25:4617 , piece_id=SEQXHIVLB2IFFS7AFBTRPT2NG7AQMYF4G4Q6X3ZDAC253J45GVMA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.104424257 (592h48m2.975507302s) [noise=true , address=145.79.227.39:4013 , piece_id=PPIXK4GRVLCBMBWKXESUK6OOV75QOPY3N7CDMKYEXJAAZXIZMKCA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.163414565 (592h48m3.03449761s) [noise=true , address=145.79.227.13:4802 , piece_id=DTYJK3TJBATMICP7CIKYJUMOGTKJDAOBENZTXN2RQASNH7E5DXSA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.165365981 (592h48m3.036449016s) [noise=true , address=145.79.227.41:4086 , piece_id=TF63E7J6W7R65MMLD47IXWSUP3DPKS2AOJ3VDIGMYN2HBKBFJGUQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.177755521 (592h48m3.048838566s) [noise=true , address=145.79.227.14:4838 , piece_id=BTGTBI6Y53DFPDMFJTFV2AOECHL3YSHNAH2AV4AKSXXOVYOBFZHA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.178258179 (592h48m3.049341224s) [noise=true , address=145.79.227.23:4547 , piece_id=W34NASX5NCYNXTDS3V7QNYE3CA7XB5WVGXWPYPVWF5JJ4NOZIIDA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.195296913 (592h48m3.066379958s) [noise=true , address=145.79.227.52:4447 , piece_id=YHBKCKPPWKZIDUI2RSZ64Y7TEWPU23L7CFFX34KR63T3AU7O4FRA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.300032047 (592h48m3.171115093s) [noise=true , address=77.90.31.14:4662 , piece_id=R7URHPL2SYNQOUVRIF3DNKYIQW2AXKDSXROHJSBBNQKVN2IX7YAA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.432987406 (592h48m3.304070451s) [noise=true , address=45.91.116.5:4298 , piece_id=GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.524295082 (592h48m3.395378127s) [noise=true , address=45.91.116.14:4192 , piece_id=POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.532208229 (592h48m3.403291274s) [noise=true , address=45.91.116.11:4119 , piece_id=NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.532228268 (592h48m3.403311303s) [seg_pos={0 0}, upload=519699]: Segment reached optimal threshold of 13 pieces. Dec 30 08:17:03.532433178 (592h48m3.403516223s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 08:17:03.532443237 (592h48m3.403526272s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532597912 (592h48m3.403680947s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 08:17:03.532605556 (592h48m3.403688592s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532899436 (592h48m3.403982481s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection Dec 30 08:17:03.532919434 (592h48m3.404002479s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532977074 (592h48m3.404060109s) [seg_pos={0 0}, upload=519699]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:17:03.533149371 (592h48m3.404232416s) [upload=519699]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:17:03.533157207 (592h48m3.404240242s) [upload=519699]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:17:03.817060166 (592h48m3.688143211s) [upload=519699]: Done uploading segments... Dec 30 08:17:03.817089281 (592h48m3.688172326s) [upload=519699]: Upload finished. err: <nil> Duration: 5.979204902s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X..................X...] duration: 3.574136569s trace: https://jaeger.dev.storj.io/trace/7a032e32a5763d1a FAIL (2025-12-30 08:18:38 UTC) orbiter warning: context deadline exceeded Duration: 3.57415814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X................X...............] duration: 44.531817571s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:18:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 44.531837539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X..................X..] duration: 3.480593226s trace: https://jaeger.dev.storj.io/trace/7a032e32a5763d1a FAIL (2025-12-30 08:18:16 UTC) orbiter warning: context deadline exceeded Duration: 3.480622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....XX.XX..................X...X..X..X..XX.X.X...XX.XX...X.] duration: 6.039153643s trace: https://jaeger.dev.storj.io/trace/20249d585c5f6db FAIL (2025-12-30 08:18:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA) context deadline exceeded ======LOGS====== Dec 30 08:17:00.846789529 (592h48m0.717872574s) [upload=519699]: Starting upload Dec 30 08:17:00.846881204 (592h48m0.717964249s) [upload=519699]: Done starting upload Dec 30 08:17:00.846963620 (592h48m0.718046665s) [upload=519699]: Uploading segments... Dec 30 08:17:00.847283640 (592h48m0.718366685s) [upload=519699]: Got next segment. Inline: false Dec 30 08:17:00.847371628 (592h48m0.718454673s) [upload=519699]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:17:00.847377619 (592h48m0.718460664s) [upload=519699]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:17:01.091255865 (592h48m0.96233891s) [seg_pos={0 0}, upload=519699]: Begin upload segment... Dec 30 08:17:01.091316200 (592h48m0.962399235s) [seg_pos={0 0}, upload=519699]: Done begin upload segment. Dec 30 08:17:01.091353702 (592h48m0.962436767s) [noise=true , address=145.79.227.41:4086 , piece_id=TF63E7J6W7R65MMLD47IXWSUP3DPKS2AOJ3VDIGMYN2HBKBFJGUQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.091424436 (592h48m0.962507481s) [noise=true , address=45.91.116.11:4119 , piece_id=NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.092107618 (592h48m0.963190663s) [noise=true , address=145.79.227.13:4802 , piece_id=DTYJK3TJBATMICP7CIKYJUMOGTKJDAOBENZTXN2RQASNH7E5DXSA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.092584986 (592h48m0.963668031s) [noise=true , address=145.79.227.23:4547 , piece_id=W34NASX5NCYNXTDS3V7QNYE3CA7XB5WVGXWPYPVWF5JJ4NOZIIDA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093019474 (592h48m0.964102519s) [noise=true , address=45.91.116.14:4192 , piece_id=POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093486252 (592h48m0.964569287s) [noise=true , address=145.79.227.3:4150 , piece_id=G7UZ2RHTY2BBFTTJRRCFKIO7FCUEJJWCERLU3BPFM2OMYQ64CD4A, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093976446 (592h48m0.965059491s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.094485345 (592h48m0.96556838s) [noise=true , address=145.79.227.14:4838 , piece_id=BTGTBI6Y53DFPDMFJTFV2AOECHL3YSHNAH2AV4AKSXXOVYOBFZHA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.095153427 (592h48m0.966236482s) [noise=true , address=145.79.227.25:4617 , piece_id=SEQXHIVLB2IFFS7AFBTRPT2NG7AQMYF4G4Q6X3ZDAC253J45GVMA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.095714736 (592h48m0.966797781s) [noise=true , address=145.79.227.39:4013 , piece_id=PPIXK4GRVLCBMBWKXESUK6OOV75QOPY3N7CDMKYEXJAAZXIZMKCA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.096269773 (592h48m0.967352818s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.096830851 (592h48m0.967913886s) [noise=true , address=145.79.227.10:4401 , piece_id=INBNWHF5KY5VKN7KRGAWA73MBU7JW7S22C4DK7BFAAQ3VTZIDGJQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.097331084 (592h48m0.968414129s) [noise=true , address=45.91.116.5:4298 , piece_id=GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.098069299 (592h48m0.969152375s) [noise=true , address=77.90.31.14:4662 , piece_id=R7URHPL2SYNQOUVRIF3DNKYIQW2AXKDSXROHJSBBNQKVN2IX7YAA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.098662018 (592h48m0.969745053s) [noise=true , address=145.79.227.52:4447 , piece_id=YHBKCKPPWKZIDUI2RSZ64Y7TEWPU23L7CFFX34KR63T3AU7O4FRA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.100199015 (592h48m0.97128205s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:02.800462773 (592h48m2.671545818s) [upload=519699]: Next returned nil segment Dec 30 08:17:02.800482280 (592h48m2.671565325s) [upload=519699]: Waiting for error group managing segments... Dec 30 08:17:03.079896141 (592h48m2.950979197s) [noise=true , address=145.79.227.3:4150 , piece_id=G7UZ2RHTY2BBFTTJRRCFKIO7FCUEJJWCERLU3BPFM2OMYQ64CD4A, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.091948050 (592h48m2.963031105s) [noise=true , address=145.79.227.10:4401 , piece_id=INBNWHF5KY5VKN7KRGAWA73MBU7JW7S22C4DK7BFAAQ3VTZIDGJQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.095491658 (592h48m2.966574703s) [noise=true , address=145.79.227.25:4617 , piece_id=SEQXHIVLB2IFFS7AFBTRPT2NG7AQMYF4G4Q6X3ZDAC253J45GVMA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.104424257 (592h48m2.975507302s) [noise=true , address=145.79.227.39:4013 , piece_id=PPIXK4GRVLCBMBWKXESUK6OOV75QOPY3N7CDMKYEXJAAZXIZMKCA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.163414565 (592h48m3.03449761s) [noise=true , address=145.79.227.13:4802 , piece_id=DTYJK3TJBATMICP7CIKYJUMOGTKJDAOBENZTXN2RQASNH7E5DXSA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.165365981 (592h48m3.036449016s) [noise=true , address=145.79.227.41:4086 , piece_id=TF63E7J6W7R65MMLD47IXWSUP3DPKS2AOJ3VDIGMYN2HBKBFJGUQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.177755521 (592h48m3.048838566s) [noise=true , address=145.79.227.14:4838 , piece_id=BTGTBI6Y53DFPDMFJTFV2AOECHL3YSHNAH2AV4AKSXXOVYOBFZHA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.178258179 (592h48m3.049341224s) [noise=true , address=145.79.227.23:4547 , piece_id=W34NASX5NCYNXTDS3V7QNYE3CA7XB5WVGXWPYPVWF5JJ4NOZIIDA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.195296913 (592h48m3.066379958s) [noise=true , address=145.79.227.52:4447 , piece_id=YHBKCKPPWKZIDUI2RSZ64Y7TEWPU23L7CFFX34KR63T3AU7O4FRA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.300032047 (592h48m3.171115093s) [noise=true , address=77.90.31.14:4662 , piece_id=R7URHPL2SYNQOUVRIF3DNKYIQW2AXKDSXROHJSBBNQKVN2IX7YAA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.432987406 (592h48m3.304070451s) [noise=true , address=45.91.116.5:4298 , piece_id=GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.524295082 (592h48m3.395378127s) [noise=true , address=45.91.116.14:4192 , piece_id=POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.532208229 (592h48m3.403291274s) [noise=true , address=45.91.116.11:4119 , piece_id=NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.532228268 (592h48m3.403311303s) [seg_pos={0 0}, upload=519699]: Segment reached optimal threshold of 13 pieces. Dec 30 08:17:03.532433178 (592h48m3.403516223s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 08:17:03.532443237 (592h48m3.403526272s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532597912 (592h48m3.403680947s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 08:17:03.532605556 (592h48m3.403688592s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532899436 (592h48m3.403982481s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection Dec 30 08:17:03.532919434 (592h48m3.404002479s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532977074 (592h48m3.404060109s) [seg_pos={0 0}, upload=519699]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:17:03.533149371 (592h48m3.404232416s) [upload=519699]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:17:03.533157207 (592h48m3.404240242s) [upload=519699]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:17:03.817060166 (592h48m3.688143211s) [upload=519699]: Done uploading segments... Dec 30 08:17:03.817089281 (592h48m3.688172326s) [upload=519699]: Upload finished. err: <nil> Duration: 6.039185775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X................X..............] duration: 32.054205231s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:17:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 32.054221201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X....XX.XX..................X...X..X..X..XX.X.X...XX.XX...X] duration: 30.002708053s trace: https://jaeger.dev.storj.io/trace/20249d585c5f6db FAIL (2025-12-30 08:17:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA) context deadline exceeded ======LOGS====== Dec 30 08:17:00.846789529 (592h48m0.717872574s) [upload=519699]: Starting upload Dec 30 08:17:00.846881204 (592h48m0.717964249s) [upload=519699]: Done starting upload Dec 30 08:17:00.846963620 (592h48m0.718046665s) [upload=519699]: Uploading segments... Dec 30 08:17:00.847283640 (592h48m0.718366685s) [upload=519699]: Got next segment. Inline: false Dec 30 08:17:00.847371628 (592h48m0.718454673s) [upload=519699]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:17:00.847377619 (592h48m0.718460664s) [upload=519699]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:17:01.091255865 (592h48m0.96233891s) [seg_pos={0 0}, upload=519699]: Begin upload segment... Dec 30 08:17:01.091316200 (592h48m0.962399235s) [seg_pos={0 0}, upload=519699]: Done begin upload segment. Dec 30 08:17:01.091353702 (592h48m0.962436767s) [noise=true , address=145.79.227.41:4086 , piece_id=TF63E7J6W7R65MMLD47IXWSUP3DPKS2AOJ3VDIGMYN2HBKBFJGUQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.091424436 (592h48m0.962507481s) [noise=true , address=45.91.116.11:4119 , piece_id=NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.092107618 (592h48m0.963190663s) [noise=true , address=145.79.227.13:4802 , piece_id=DTYJK3TJBATMICP7CIKYJUMOGTKJDAOBENZTXN2RQASNH7E5DXSA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.092584986 (592h48m0.963668031s) [noise=true , address=145.79.227.23:4547 , piece_id=W34NASX5NCYNXTDS3V7QNYE3CA7XB5WVGXWPYPVWF5JJ4NOZIIDA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093019474 (592h48m0.964102519s) [noise=true , address=45.91.116.14:4192 , piece_id=POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093486252 (592h48m0.964569287s) [noise=true , address=145.79.227.3:4150 , piece_id=G7UZ2RHTY2BBFTTJRRCFKIO7FCUEJJWCERLU3BPFM2OMYQ64CD4A, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.093976446 (592h48m0.965059491s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.094485345 (592h48m0.96556838s) [noise=true , address=145.79.227.14:4838 , piece_id=BTGTBI6Y53DFPDMFJTFV2AOECHL3YSHNAH2AV4AKSXXOVYOBFZHA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.095153427 (592h48m0.966236482s) [noise=true , address=145.79.227.25:4617 , piece_id=SEQXHIVLB2IFFS7AFBTRPT2NG7AQMYF4G4Q6X3ZDAC253J45GVMA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.095714736 (592h48m0.966797781s) [noise=true , address=145.79.227.39:4013 , piece_id=PPIXK4GRVLCBMBWKXESUK6OOV75QOPY3N7CDMKYEXJAAZXIZMKCA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.096269773 (592h48m0.967352818s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.096830851 (592h48m0.967913886s) [noise=true , address=145.79.227.10:4401 , piece_id=INBNWHF5KY5VKN7KRGAWA73MBU7JW7S22C4DK7BFAAQ3VTZIDGJQ, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.097331084 (592h48m0.968414129s) [noise=true , address=45.91.116.5:4298 , piece_id=GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.098069299 (592h48m0.969152375s) [noise=true , address=77.90.31.14:4662 , piece_id=R7URHPL2SYNQOUVRIF3DNKYIQW2AXKDSXROHJSBBNQKVN2IX7YAA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.098662018 (592h48m0.969745053s) [noise=true , address=145.79.227.52:4447 , piece_id=YHBKCKPPWKZIDUI2RSZ64Y7TEWPU23L7CFFX34KR63T3AU7O4FRA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:01.100199015 (592h48m0.97128205s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Uploading piece... Dec 30 08:17:02.800462773 (592h48m2.671545818s) [upload=519699]: Next returned nil segment Dec 30 08:17:02.800482280 (592h48m2.671565325s) [upload=519699]: Waiting for error group managing segments... Dec 30 08:17:03.079896141 (592h48m2.950979197s) [noise=true , address=145.79.227.3:4150 , piece_id=G7UZ2RHTY2BBFTTJRRCFKIO7FCUEJJWCERLU3BPFM2OMYQ64CD4A, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.091948050 (592h48m2.963031105s) [noise=true , address=145.79.227.10:4401 , piece_id=INBNWHF5KY5VKN7KRGAWA73MBU7JW7S22C4DK7BFAAQ3VTZIDGJQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.095491658 (592h48m2.966574703s) [noise=true , address=145.79.227.25:4617 , piece_id=SEQXHIVLB2IFFS7AFBTRPT2NG7AQMYF4G4Q6X3ZDAC253J45GVMA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.104424257 (592h48m2.975507302s) [noise=true , address=145.79.227.39:4013 , piece_id=PPIXK4GRVLCBMBWKXESUK6OOV75QOPY3N7CDMKYEXJAAZXIZMKCA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.163414565 (592h48m3.03449761s) [noise=true , address=145.79.227.13:4802 , piece_id=DTYJK3TJBATMICP7CIKYJUMOGTKJDAOBENZTXN2RQASNH7E5DXSA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.165365981 (592h48m3.036449016s) [noise=true , address=145.79.227.41:4086 , piece_id=TF63E7J6W7R65MMLD47IXWSUP3DPKS2AOJ3VDIGMYN2HBKBFJGUQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.177755521 (592h48m3.048838566s) [noise=true , address=145.79.227.14:4838 , piece_id=BTGTBI6Y53DFPDMFJTFV2AOECHL3YSHNAH2AV4AKSXXOVYOBFZHA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.178258179 (592h48m3.049341224s) [noise=true , address=145.79.227.23:4547 , piece_id=W34NASX5NCYNXTDS3V7QNYE3CA7XB5WVGXWPYPVWF5JJ4NOZIIDA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.195296913 (592h48m3.066379958s) [noise=true , address=145.79.227.52:4447 , piece_id=YHBKCKPPWKZIDUI2RSZ64Y7TEWPU23L7CFFX34KR63T3AU7O4FRA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.300032047 (592h48m3.171115093s) [noise=true , address=77.90.31.14:4662 , piece_id=R7URHPL2SYNQOUVRIF3DNKYIQW2AXKDSXROHJSBBNQKVN2IX7YAA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.432987406 (592h48m3.304070451s) [noise=true , address=45.91.116.5:4298 , piece_id=GITGXPSM6Z5Z3BI6MJFMAE4OMXZEJSECHVEGNAUNZYUY4CNGXKNA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.524295082 (592h48m3.395378127s) [noise=true , address=45.91.116.14:4192 , piece_id=POGHQGXAEEL6AE4V6HMB3YASKQWRI5Y2A6EJXXHRTUU7ZT2KOENA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.532208229 (592h48m3.403291274s) [noise=true , address=45.91.116.11:4119 , piece_id=NRREBFBZUZWMPCEORPL3JQDBYCAMKN73GL576D6OMLJUBSS5P3HQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: <nil> Dec 30 08:17:03.532228268 (592h48m3.403311303s) [seg_pos={0 0}, upload=519699]: Segment reached optimal threshold of 13 pieces. Dec 30 08:17:03.532433178 (592h48m3.403516223s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 08:17:03.532443237 (592h48m3.403526272s) [noise=true , address=45.91.116.13:4156 , piece_id=JHOZDM6SO3Q7A5E6BZW5GSZFQWGW2F2SPWASVPSYPPTMLIL5FEOA, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532597912 (592h48m3.403680947s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 08:17:03.532605556 (592h48m3.403688592s) [noise=true , address=145.79.227.29:4943 , piece_id=ZNREBHNKHDGYZGOIKTZNXYWJK72D6ZW2L66IJ6E5FCZBIBOFCWTQ, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532899436 (592h48m3.403982481s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52656->77.90.31.19:4841: use of closed network connection Dec 30 08:17:03.532919434 (592h48m3.404002479s) [noise=true , address=77.90.31.19:4841 , piece_id=OIBDT22W4YE5BO2ZOINXUM45NYICLKDJUPK5BBSEDTJA7S6RTZUQ, seg_pos={0 0}, upload=519699]: Long tail cancellation due to optimal threshold Dec 30 08:17:03.532977074 (592h48m3.404060109s) [seg_pos={0 0}, upload=519699]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:17:03.533149371 (592h48m3.404232416s) [upload=519699]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:17:03.533157207 (592h48m3.404240242s) [upload=519699]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:17:03.817060166 (592h48m3.688143211s) [upload=519699]: Done uploading segments... Dec 30 08:17:03.817089281 (592h48m3.688172326s) [upload=519699]: Upload finished. err: <nil> Duration: 30.002730797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X................X.............] duration: 46.232221405s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:17:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 46.232245993s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X..................X.] duration: 3.609976764s trace: https://jaeger.dev.storj.io/trace/7a032e32a5763d1a FAIL (2025-12-30 08:17:11 UTC) orbiter warning: context deadline exceeded Duration: 3.610003483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X..................X] duration: 5.001638711s trace: https://jaeger.dev.storj.io/trace/7a032e32a5763d1a FAIL (2025-12-30 08:16:34 UTC) orbiter warning: context deadline exceeded Duration: 5.001661514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X....XX.XX..................X...X..X..X..XX.X.X...XX.XX...] duration: 5.89157595s trace: https://jaeger.dev.storj.io/trace/3b704cf49b51f88f FAIL (2025-12-30 08:16:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 08:14:25.660520365 (592h45m25.53160341s) [upload=519659]: Starting upload Dec 30 08:14:25.660581702 (592h45m25.531664747s) [upload=519659]: Done starting upload Dec 30 08:14:25.660696059 (592h45m25.531779094s) [upload=519659]: Uploading segments... Dec 30 08:14:25.660761003 (592h45m25.531844038s) [upload=519659]: Got next segment. Inline: false Dec 30 08:14:25.660787925 (592h45m25.53187096s) [upload=519659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:14:25.660792784 (592h45m25.531875829s) [upload=519659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:14:25.906363599 (592h45m25.777446644s) [seg_pos={0 0}, upload=519659]: Begin upload segment... Dec 30 08:14:25.906424686 (592h45m25.777507721s) [seg_pos={0 0}, upload=519659]: Done begin upload segment. Dec 30 08:14:25.906437881 (592h45m25.777520926s) [noise=true , address=145.79.227.46:4602 , piece_id=LZULKLURRHRCBBALYU6DK2JZUN6DEC42VJUWYYQ7I26NSWQ4GOBQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.906450364 (592h45m25.777533399s) [noise=true , address=77.90.31.18:4805 , piece_id=SDTNLVREX26UFYEWQUDV6EX4YSNZJISGSQFS7ZUYFBKVGRBXSRZQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.906557659 (592h45m25.777640704s) [noise=true , address=45.91.116.12:4225 , piece_id=4RJ2HL5VOOVHDWFUQOWRNSND7EJ2D2MIG53R6X5SV4HBYNE2ORBQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.907220651 (592h45m25.778303686s) [noise=true , address=145.79.227.16:4910 , piece_id=MBMAC3EPRXNMPUYAASMUFH2MUWW7ZECYRXTGAZLJFAQXXTQJU3EA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.907718329 (592h45m25.778801374s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908056532 (592h45m25.779139567s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908492062 (592h45m25.779575107s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908886593 (592h45m25.779969638s) [noise=true , address=145.79.227.9:4367 , piece_id=C4KPWR5FTHQJZQ2NCGGHFR3YH5JMOLJ35LVCEDLUULLY24DRH2OA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.909291765 (592h45m25.78037481s) [noise=true , address=145.79.227.22:4509 , piece_id=CPVKNKKLDRQSHRWBZPW5GNC5N3ZQYIBQDUMJMOSPMNDW3KFZQQUQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.909859756 (592h45m25.780942791s) [noise=true , address=145.79.227.11:4731 , piece_id=KPVRZSKWZCR3QAAK6IX7ORQXWJR33FBZDEEFTUZX5CS2IAPK4ILA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.911917725 (592h45m25.78300077s) [noise=true , address=145.79.227.10:4403 , piece_id=RCUNDOMOLAY4U37K4QONKBRUE62SYJ43GTAKVKQITLUAGECOULUA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.913587014 (592h45m25.784670049s) [noise=true , address=145.79.227.14:4839 , piece_id=BWUAORX5JM4B7R32R32GTSMFSDC5RUM7YDDV5Z67BRLXKBCXIW4A, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.916790874 (592h45m25.787873929s) [noise=true , address=45.91.116.13:4154 , piece_id=R3NPUQ5PXI4L5Q2YR2O7DMG7ZVJZ4GUCYAUQ5BCL7MJ6YJJ44O7A, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.919099380 (592h45m25.790182425s) [noise=true , address=145.79.227.41:4087 , piece_id=KPOMWQGIFBR5VF2BQTKMG7XZAI5RZQGNM7KQ2WDLZILPST627WTQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.921528285 (592h45m25.79261132s) [noise=true , address=145.79.227.29:4943 , piece_id=U4CSBL3CA2IJHPKNFX5VRG2NMCNZ2PV4MY6OC44E5YXSWS6CKPUA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.923345124 (592h45m25.794428159s) [noise=true , address=145.79.227.8:4331 , piece_id=JQE2P5P322EACXUGWNHASSV6TZMIP5RWMOZMKR7MIMHBR43WWESQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:27.601675641 (592h45m27.472758686s) [upload=519659]: Next returned nil segment Dec 30 08:14:27.601722752 (592h45m27.472805797s) [upload=519659]: Waiting for error group managing segments... Dec 30 08:14:27.894739231 (592h45m27.765822276s) [noise=true , address=145.79.227.46:4602 , piece_id=LZULKLURRHRCBBALYU6DK2JZUN6DEC42VJUWYYQ7I26NSWQ4GOBQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.896830954 (592h45m27.767913999s) [noise=true , address=145.79.227.10:4403 , piece_id=RCUNDOMOLAY4U37K4QONKBRUE62SYJ43GTAKVKQITLUAGECOULUA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.908346201 (592h45m27.779429237s) [noise=true , address=145.79.227.9:4367 , piece_id=C4KPWR5FTHQJZQ2NCGGHFR3YH5JMOLJ35LVCEDLUULLY24DRH2OA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.910303097 (592h45m27.781386143s) [noise=true , address=145.79.227.16:4910 , piece_id=MBMAC3EPRXNMPUYAASMUFH2MUWW7ZECYRXTGAZLJFAQXXTQJU3EA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.911969350 (592h45m27.783052396s) [noise=true , address=145.79.227.11:4731 , piece_id=KPVRZSKWZCR3QAAK6IX7ORQXWJR33FBZDEEFTUZX5CS2IAPK4ILA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.931037770 (592h45m27.802120805s) [noise=true , address=145.79.227.14:4839 , piece_id=BWUAORX5JM4B7R32R32GTSMFSDC5RUM7YDDV5Z67BRLXKBCXIW4A, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.931205659 (592h45m27.802288694s) [noise=true , address=145.79.227.22:4509 , piece_id=CPVKNKKLDRQSHRWBZPW5GNC5N3ZQYIBQDUMJMOSPMNDW3KFZQQUQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.947385048 (592h45m27.818468083s) [noise=true , address=145.79.227.29:4943 , piece_id=U4CSBL3CA2IJHPKNFX5VRG2NMCNZ2PV4MY6OC44E5YXSWS6CKPUA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.000304415 (592h45m27.87138747s) [noise=true , address=145.79.227.8:4331 , piece_id=JQE2P5P322EACXUGWNHASSV6TZMIP5RWMOZMKR7MIMHBR43WWESQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.014732849 (592h45m27.885815894s) [noise=true , address=145.79.227.41:4087 , piece_id=KPOMWQGIFBR5VF2BQTKMG7XZAI5RZQGNM7KQ2WDLZILPST627WTQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.171983078 (592h45m28.043066113s) [noise=true , address=77.90.31.18:4805 , piece_id=SDTNLVREX26UFYEWQUDV6EX4YSNZJISGSQFS7ZUYFBKVGRBXSRZQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.213841073 (592h45m28.084924128s) [noise=true , address=45.91.116.13:4154 , piece_id=R3NPUQ5PXI4L5Q2YR2O7DMG7ZVJZ4GUCYAUQ5BCL7MJ6YJJ44O7A, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.222817134 (592h45m28.093900179s) [noise=true , address=45.91.116.12:4225 , piece_id=4RJ2HL5VOOVHDWFUQOWRNSND7EJ2D2MIG53R6X5SV4HBYNE2ORBQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.222843214 (592h45m28.093926249s) [seg_pos={0 0}, upload=519659]: Segment reached optimal threshold of 13 pieces. Dec 30 08:14:28.223173752 (592h45m28.094256798s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 30 08:14:28.223182590 (592h45m28.094265635s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223546122 (592h45m28.094629167s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection Dec 30 08:14:28.223557032 (592h45m28.094640077s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223746033 (592h45m28.094829068s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 08:14:28.223752645 (592h45m28.09483568s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223768846 (592h45m28.094851881s) [seg_pos={0 0}, upload=519659]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:14:28.223839430 (592h45m28.094922465s) [upload=519659]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:14:28.223845112 (592h45m28.094928147s) [upload=519659]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:14:28.507406377 (592h45m28.378489422s) [upload=519659]: Done uploading segments... Dec 30 08:14:28.507431966 (592h45m28.378515011s) [upload=519659]: Upload finished. err: <nil> Duration: 5.891606769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X................X............] duration: 36.19827207s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:16:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 36.198300825s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X..................] duration: 3.667951969s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:15:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.667983167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X....XX.XX..................X...X..X..X..XX.X.X...XX.XX..] duration: 5.7455169s trace: https://jaeger.dev.storj.io/trace/3b704cf49b51f88f FAIL (2025-12-30 08:15:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 08:14:25.660520365 (592h45m25.53160341s) [upload=519659]: Starting upload Dec 30 08:14:25.660581702 (592h45m25.531664747s) [upload=519659]: Done starting upload Dec 30 08:14:25.660696059 (592h45m25.531779094s) [upload=519659]: Uploading segments... Dec 30 08:14:25.660761003 (592h45m25.531844038s) [upload=519659]: Got next segment. Inline: false Dec 30 08:14:25.660787925 (592h45m25.53187096s) [upload=519659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:14:25.660792784 (592h45m25.531875829s) [upload=519659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:14:25.906363599 (592h45m25.777446644s) [seg_pos={0 0}, upload=519659]: Begin upload segment... Dec 30 08:14:25.906424686 (592h45m25.777507721s) [seg_pos={0 0}, upload=519659]: Done begin upload segment. Dec 30 08:14:25.906437881 (592h45m25.777520926s) [noise=true , address=145.79.227.46:4602 , piece_id=LZULKLURRHRCBBALYU6DK2JZUN6DEC42VJUWYYQ7I26NSWQ4GOBQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.906450364 (592h45m25.777533399s) [noise=true , address=77.90.31.18:4805 , piece_id=SDTNLVREX26UFYEWQUDV6EX4YSNZJISGSQFS7ZUYFBKVGRBXSRZQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.906557659 (592h45m25.777640704s) [noise=true , address=45.91.116.12:4225 , piece_id=4RJ2HL5VOOVHDWFUQOWRNSND7EJ2D2MIG53R6X5SV4HBYNE2ORBQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.907220651 (592h45m25.778303686s) [noise=true , address=145.79.227.16:4910 , piece_id=MBMAC3EPRXNMPUYAASMUFH2MUWW7ZECYRXTGAZLJFAQXXTQJU3EA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.907718329 (592h45m25.778801374s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908056532 (592h45m25.779139567s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908492062 (592h45m25.779575107s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908886593 (592h45m25.779969638s) [noise=true , address=145.79.227.9:4367 , piece_id=C4KPWR5FTHQJZQ2NCGGHFR3YH5JMOLJ35LVCEDLUULLY24DRH2OA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.909291765 (592h45m25.78037481s) [noise=true , address=145.79.227.22:4509 , piece_id=CPVKNKKLDRQSHRWBZPW5GNC5N3ZQYIBQDUMJMOSPMNDW3KFZQQUQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.909859756 (592h45m25.780942791s) [noise=true , address=145.79.227.11:4731 , piece_id=KPVRZSKWZCR3QAAK6IX7ORQXWJR33FBZDEEFTUZX5CS2IAPK4ILA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.911917725 (592h45m25.78300077s) [noise=true , address=145.79.227.10:4403 , piece_id=RCUNDOMOLAY4U37K4QONKBRUE62SYJ43GTAKVKQITLUAGECOULUA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.913587014 (592h45m25.784670049s) [noise=true , address=145.79.227.14:4839 , piece_id=BWUAORX5JM4B7R32R32GTSMFSDC5RUM7YDDV5Z67BRLXKBCXIW4A, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.916790874 (592h45m25.787873929s) [noise=true , address=45.91.116.13:4154 , piece_id=R3NPUQ5PXI4L5Q2YR2O7DMG7ZVJZ4GUCYAUQ5BCL7MJ6YJJ44O7A, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.919099380 (592h45m25.790182425s) [noise=true , address=145.79.227.41:4087 , piece_id=KPOMWQGIFBR5VF2BQTKMG7XZAI5RZQGNM7KQ2WDLZILPST627WTQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.921528285 (592h45m25.79261132s) [noise=true , address=145.79.227.29:4943 , piece_id=U4CSBL3CA2IJHPKNFX5VRG2NMCNZ2PV4MY6OC44E5YXSWS6CKPUA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.923345124 (592h45m25.794428159s) [noise=true , address=145.79.227.8:4331 , piece_id=JQE2P5P322EACXUGWNHASSV6TZMIP5RWMOZMKR7MIMHBR43WWESQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:27.601675641 (592h45m27.472758686s) [upload=519659]: Next returned nil segment Dec 30 08:14:27.601722752 (592h45m27.472805797s) [upload=519659]: Waiting for error group managing segments... Dec 30 08:14:27.894739231 (592h45m27.765822276s) [noise=true , address=145.79.227.46:4602 , piece_id=LZULKLURRHRCBBALYU6DK2JZUN6DEC42VJUWYYQ7I26NSWQ4GOBQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.896830954 (592h45m27.767913999s) [noise=true , address=145.79.227.10:4403 , piece_id=RCUNDOMOLAY4U37K4QONKBRUE62SYJ43GTAKVKQITLUAGECOULUA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.908346201 (592h45m27.779429237s) [noise=true , address=145.79.227.9:4367 , piece_id=C4KPWR5FTHQJZQ2NCGGHFR3YH5JMOLJ35LVCEDLUULLY24DRH2OA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.910303097 (592h45m27.781386143s) [noise=true , address=145.79.227.16:4910 , piece_id=MBMAC3EPRXNMPUYAASMUFH2MUWW7ZECYRXTGAZLJFAQXXTQJU3EA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.911969350 (592h45m27.783052396s) [noise=true , address=145.79.227.11:4731 , piece_id=KPVRZSKWZCR3QAAK6IX7ORQXWJR33FBZDEEFTUZX5CS2IAPK4ILA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.931037770 (592h45m27.802120805s) [noise=true , address=145.79.227.14:4839 , piece_id=BWUAORX5JM4B7R32R32GTSMFSDC5RUM7YDDV5Z67BRLXKBCXIW4A, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.931205659 (592h45m27.802288694s) [noise=true , address=145.79.227.22:4509 , piece_id=CPVKNKKLDRQSHRWBZPW5GNC5N3ZQYIBQDUMJMOSPMNDW3KFZQQUQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.947385048 (592h45m27.818468083s) [noise=true , address=145.79.227.29:4943 , piece_id=U4CSBL3CA2IJHPKNFX5VRG2NMCNZ2PV4MY6OC44E5YXSWS6CKPUA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.000304415 (592h45m27.87138747s) [noise=true , address=145.79.227.8:4331 , piece_id=JQE2P5P322EACXUGWNHASSV6TZMIP5RWMOZMKR7MIMHBR43WWESQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.014732849 (592h45m27.885815894s) [noise=true , address=145.79.227.41:4087 , piece_id=KPOMWQGIFBR5VF2BQTKMG7XZAI5RZQGNM7KQ2WDLZILPST627WTQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.171983078 (592h45m28.043066113s) [noise=true , address=77.90.31.18:4805 , piece_id=SDTNLVREX26UFYEWQUDV6EX4YSNZJISGSQFS7ZUYFBKVGRBXSRZQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.213841073 (592h45m28.084924128s) [noise=true , address=45.91.116.13:4154 , piece_id=R3NPUQ5PXI4L5Q2YR2O7DMG7ZVJZ4GUCYAUQ5BCL7MJ6YJJ44O7A, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.222817134 (592h45m28.093900179s) [noise=true , address=45.91.116.12:4225 , piece_id=4RJ2HL5VOOVHDWFUQOWRNSND7EJ2D2MIG53R6X5SV4HBYNE2ORBQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.222843214 (592h45m28.093926249s) [seg_pos={0 0}, upload=519659]: Segment reached optimal threshold of 13 pieces. Dec 30 08:14:28.223173752 (592h45m28.094256798s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 30 08:14:28.223182590 (592h45m28.094265635s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223546122 (592h45m28.094629167s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection Dec 30 08:14:28.223557032 (592h45m28.094640077s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223746033 (592h45m28.094829068s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 08:14:28.223752645 (592h45m28.09483568s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223768846 (592h45m28.094851881s) [seg_pos={0 0}, upload=519659]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:14:28.223839430 (592h45m28.094922465s) [upload=519659]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:14:28.223845112 (592h45m28.094928147s) [upload=519659]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:14:28.507406377 (592h45m28.378489422s) [upload=519659]: Done uploading segments... Dec 30 08:14:28.507431966 (592h45m28.378515011s) [upload=519659]: Upload finished. err: <nil> Duration: 5.745543159s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X.................] duration: 3.458000064s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:15:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.458027426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX..X....XX.XX..................X...X..X..X..XX.X.X...XX.XX.] duration: 5.856212084s trace: https://jaeger.dev.storj.io/trace/3b704cf49b51f88f FAIL (2025-12-30 08:15:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 08:14:25.660520365 (592h45m25.53160341s) [upload=519659]: Starting upload Dec 30 08:14:25.660581702 (592h45m25.531664747s) [upload=519659]: Done starting upload Dec 30 08:14:25.660696059 (592h45m25.531779094s) [upload=519659]: Uploading segments... Dec 30 08:14:25.660761003 (592h45m25.531844038s) [upload=519659]: Got next segment. Inline: false Dec 30 08:14:25.660787925 (592h45m25.53187096s) [upload=519659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:14:25.660792784 (592h45m25.531875829s) [upload=519659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:14:25.906363599 (592h45m25.777446644s) [seg_pos={0 0}, upload=519659]: Begin upload segment... Dec 30 08:14:25.906424686 (592h45m25.777507721s) [seg_pos={0 0}, upload=519659]: Done begin upload segment. Dec 30 08:14:25.906437881 (592h45m25.777520926s) [noise=true , address=145.79.227.46:4602 , piece_id=LZULKLURRHRCBBALYU6DK2JZUN6DEC42VJUWYYQ7I26NSWQ4GOBQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.906450364 (592h45m25.777533399s) [noise=true , address=77.90.31.18:4805 , piece_id=SDTNLVREX26UFYEWQUDV6EX4YSNZJISGSQFS7ZUYFBKVGRBXSRZQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.906557659 (592h45m25.777640704s) [noise=true , address=45.91.116.12:4225 , piece_id=4RJ2HL5VOOVHDWFUQOWRNSND7EJ2D2MIG53R6X5SV4HBYNE2ORBQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.907220651 (592h45m25.778303686s) [noise=true , address=145.79.227.16:4910 , piece_id=MBMAC3EPRXNMPUYAASMUFH2MUWW7ZECYRXTGAZLJFAQXXTQJU3EA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.907718329 (592h45m25.778801374s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908056532 (592h45m25.779139567s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908492062 (592h45m25.779575107s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908886593 (592h45m25.779969638s) [noise=true , address=145.79.227.9:4367 , piece_id=C4KPWR5FTHQJZQ2NCGGHFR3YH5JMOLJ35LVCEDLUULLY24DRH2OA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.909291765 (592h45m25.78037481s) [noise=true , address=145.79.227.22:4509 , piece_id=CPVKNKKLDRQSHRWBZPW5GNC5N3ZQYIBQDUMJMOSPMNDW3KFZQQUQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.909859756 (592h45m25.780942791s) [noise=true , address=145.79.227.11:4731 , piece_id=KPVRZSKWZCR3QAAK6IX7ORQXWJR33FBZDEEFTUZX5CS2IAPK4ILA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.911917725 (592h45m25.78300077s) [noise=true , address=145.79.227.10:4403 , piece_id=RCUNDOMOLAY4U37K4QONKBRUE62SYJ43GTAKVKQITLUAGECOULUA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.913587014 (592h45m25.784670049s) [noise=true , address=145.79.227.14:4839 , piece_id=BWUAORX5JM4B7R32R32GTSMFSDC5RUM7YDDV5Z67BRLXKBCXIW4A, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.916790874 (592h45m25.787873929s) [noise=true , address=45.91.116.13:4154 , piece_id=R3NPUQ5PXI4L5Q2YR2O7DMG7ZVJZ4GUCYAUQ5BCL7MJ6YJJ44O7A, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.919099380 (592h45m25.790182425s) [noise=true , address=145.79.227.41:4087 , piece_id=KPOMWQGIFBR5VF2BQTKMG7XZAI5RZQGNM7KQ2WDLZILPST627WTQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.921528285 (592h45m25.79261132s) [noise=true , address=145.79.227.29:4943 , piece_id=U4CSBL3CA2IJHPKNFX5VRG2NMCNZ2PV4MY6OC44E5YXSWS6CKPUA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.923345124 (592h45m25.794428159s) [noise=true , address=145.79.227.8:4331 , piece_id=JQE2P5P322EACXUGWNHASSV6TZMIP5RWMOZMKR7MIMHBR43WWESQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:27.601675641 (592h45m27.472758686s) [upload=519659]: Next returned nil segment Dec 30 08:14:27.601722752 (592h45m27.472805797s) [upload=519659]: Waiting for error group managing segments... Dec 30 08:14:27.894739231 (592h45m27.765822276s) [noise=true , address=145.79.227.46:4602 , piece_id=LZULKLURRHRCBBALYU6DK2JZUN6DEC42VJUWYYQ7I26NSWQ4GOBQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.896830954 (592h45m27.767913999s) [noise=true , address=145.79.227.10:4403 , piece_id=RCUNDOMOLAY4U37K4QONKBRUE62SYJ43GTAKVKQITLUAGECOULUA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.908346201 (592h45m27.779429237s) [noise=true , address=145.79.227.9:4367 , piece_id=C4KPWR5FTHQJZQ2NCGGHFR3YH5JMOLJ35LVCEDLUULLY24DRH2OA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.910303097 (592h45m27.781386143s) [noise=true , address=145.79.227.16:4910 , piece_id=MBMAC3EPRXNMPUYAASMUFH2MUWW7ZECYRXTGAZLJFAQXXTQJU3EA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.911969350 (592h45m27.783052396s) [noise=true , address=145.79.227.11:4731 , piece_id=KPVRZSKWZCR3QAAK6IX7ORQXWJR33FBZDEEFTUZX5CS2IAPK4ILA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.931037770 (592h45m27.802120805s) [noise=true , address=145.79.227.14:4839 , piece_id=BWUAORX5JM4B7R32R32GTSMFSDC5RUM7YDDV5Z67BRLXKBCXIW4A, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.931205659 (592h45m27.802288694s) [noise=true , address=145.79.227.22:4509 , piece_id=CPVKNKKLDRQSHRWBZPW5GNC5N3ZQYIBQDUMJMOSPMNDW3KFZQQUQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.947385048 (592h45m27.818468083s) [noise=true , address=145.79.227.29:4943 , piece_id=U4CSBL3CA2IJHPKNFX5VRG2NMCNZ2PV4MY6OC44E5YXSWS6CKPUA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.000304415 (592h45m27.87138747s) [noise=true , address=145.79.227.8:4331 , piece_id=JQE2P5P322EACXUGWNHASSV6TZMIP5RWMOZMKR7MIMHBR43WWESQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.014732849 (592h45m27.885815894s) [noise=true , address=145.79.227.41:4087 , piece_id=KPOMWQGIFBR5VF2BQTKMG7XZAI5RZQGNM7KQ2WDLZILPST627WTQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.171983078 (592h45m28.043066113s) [noise=true , address=77.90.31.18:4805 , piece_id=SDTNLVREX26UFYEWQUDV6EX4YSNZJISGSQFS7ZUYFBKVGRBXSRZQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.213841073 (592h45m28.084924128s) [noise=true , address=45.91.116.13:4154 , piece_id=R3NPUQ5PXI4L5Q2YR2O7DMG7ZVJZ4GUCYAUQ5BCL7MJ6YJJ44O7A, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.222817134 (592h45m28.093900179s) [noise=true , address=45.91.116.12:4225 , piece_id=4RJ2HL5VOOVHDWFUQOWRNSND7EJ2D2MIG53R6X5SV4HBYNE2ORBQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.222843214 (592h45m28.093926249s) [seg_pos={0 0}, upload=519659]: Segment reached optimal threshold of 13 pieces. Dec 30 08:14:28.223173752 (592h45m28.094256798s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 30 08:14:28.223182590 (592h45m28.094265635s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223546122 (592h45m28.094629167s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection Dec 30 08:14:28.223557032 (592h45m28.094640077s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223746033 (592h45m28.094829068s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 08:14:28.223752645 (592h45m28.09483568s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223768846 (592h45m28.094851881s) [seg_pos={0 0}, upload=519659]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:14:28.223839430 (592h45m28.094922465s) [upload=519659]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:14:28.223845112 (592h45m28.094928147s) [upload=519659]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:14:28.507406377 (592h45m28.378489422s) [upload=519659]: Done uploading segments... Dec 30 08:14:28.507431966 (592h45m28.378515011s) [upload=519659]: Upload finished. err: <nil> Duration: 5.856239967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX..X....XX.XX..................X...X..X..X..XX.X.X...XX.XX] duration: 30.003492419s trace: https://jaeger.dev.storj.io/trace/3b704cf49b51f88f FAIL (2025-12-30 08:14:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 08:14:25.660520365 (592h45m25.53160341s) [upload=519659]: Starting upload Dec 30 08:14:25.660581702 (592h45m25.531664747s) [upload=519659]: Done starting upload Dec 30 08:14:25.660696059 (592h45m25.531779094s) [upload=519659]: Uploading segments... Dec 30 08:14:25.660761003 (592h45m25.531844038s) [upload=519659]: Got next segment. Inline: false Dec 30 08:14:25.660787925 (592h45m25.53187096s) [upload=519659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:14:25.660792784 (592h45m25.531875829s) [upload=519659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:14:25.906363599 (592h45m25.777446644s) [seg_pos={0 0}, upload=519659]: Begin upload segment... Dec 30 08:14:25.906424686 (592h45m25.777507721s) [seg_pos={0 0}, upload=519659]: Done begin upload segment. Dec 30 08:14:25.906437881 (592h45m25.777520926s) [noise=true , address=145.79.227.46:4602 , piece_id=LZULKLURRHRCBBALYU6DK2JZUN6DEC42VJUWYYQ7I26NSWQ4GOBQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.906450364 (592h45m25.777533399s) [noise=true , address=77.90.31.18:4805 , piece_id=SDTNLVREX26UFYEWQUDV6EX4YSNZJISGSQFS7ZUYFBKVGRBXSRZQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.906557659 (592h45m25.777640704s) [noise=true , address=45.91.116.12:4225 , piece_id=4RJ2HL5VOOVHDWFUQOWRNSND7EJ2D2MIG53R6X5SV4HBYNE2ORBQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.907220651 (592h45m25.778303686s) [noise=true , address=145.79.227.16:4910 , piece_id=MBMAC3EPRXNMPUYAASMUFH2MUWW7ZECYRXTGAZLJFAQXXTQJU3EA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.907718329 (592h45m25.778801374s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908056532 (592h45m25.779139567s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908492062 (592h45m25.779575107s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.908886593 (592h45m25.779969638s) [noise=true , address=145.79.227.9:4367 , piece_id=C4KPWR5FTHQJZQ2NCGGHFR3YH5JMOLJ35LVCEDLUULLY24DRH2OA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.909291765 (592h45m25.78037481s) [noise=true , address=145.79.227.22:4509 , piece_id=CPVKNKKLDRQSHRWBZPW5GNC5N3ZQYIBQDUMJMOSPMNDW3KFZQQUQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.909859756 (592h45m25.780942791s) [noise=true , address=145.79.227.11:4731 , piece_id=KPVRZSKWZCR3QAAK6IX7ORQXWJR33FBZDEEFTUZX5CS2IAPK4ILA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.911917725 (592h45m25.78300077s) [noise=true , address=145.79.227.10:4403 , piece_id=RCUNDOMOLAY4U37K4QONKBRUE62SYJ43GTAKVKQITLUAGECOULUA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.913587014 (592h45m25.784670049s) [noise=true , address=145.79.227.14:4839 , piece_id=BWUAORX5JM4B7R32R32GTSMFSDC5RUM7YDDV5Z67BRLXKBCXIW4A, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.916790874 (592h45m25.787873929s) [noise=true , address=45.91.116.13:4154 , piece_id=R3NPUQ5PXI4L5Q2YR2O7DMG7ZVJZ4GUCYAUQ5BCL7MJ6YJJ44O7A, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.919099380 (592h45m25.790182425s) [noise=true , address=145.79.227.41:4087 , piece_id=KPOMWQGIFBR5VF2BQTKMG7XZAI5RZQGNM7KQ2WDLZILPST627WTQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.921528285 (592h45m25.79261132s) [noise=true , address=145.79.227.29:4943 , piece_id=U4CSBL3CA2IJHPKNFX5VRG2NMCNZ2PV4MY6OC44E5YXSWS6CKPUA, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:25.923345124 (592h45m25.794428159s) [noise=true , address=145.79.227.8:4331 , piece_id=JQE2P5P322EACXUGWNHASSV6TZMIP5RWMOZMKR7MIMHBR43WWESQ, seg_pos={0 0}, upload=519659]: Uploading piece... Dec 30 08:14:27.601675641 (592h45m27.472758686s) [upload=519659]: Next returned nil segment Dec 30 08:14:27.601722752 (592h45m27.472805797s) [upload=519659]: Waiting for error group managing segments... Dec 30 08:14:27.894739231 (592h45m27.765822276s) [noise=true , address=145.79.227.46:4602 , piece_id=LZULKLURRHRCBBALYU6DK2JZUN6DEC42VJUWYYQ7I26NSWQ4GOBQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.896830954 (592h45m27.767913999s) [noise=true , address=145.79.227.10:4403 , piece_id=RCUNDOMOLAY4U37K4QONKBRUE62SYJ43GTAKVKQITLUAGECOULUA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.908346201 (592h45m27.779429237s) [noise=true , address=145.79.227.9:4367 , piece_id=C4KPWR5FTHQJZQ2NCGGHFR3YH5JMOLJ35LVCEDLUULLY24DRH2OA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.910303097 (592h45m27.781386143s) [noise=true , address=145.79.227.16:4910 , piece_id=MBMAC3EPRXNMPUYAASMUFH2MUWW7ZECYRXTGAZLJFAQXXTQJU3EA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.911969350 (592h45m27.783052396s) [noise=true , address=145.79.227.11:4731 , piece_id=KPVRZSKWZCR3QAAK6IX7ORQXWJR33FBZDEEFTUZX5CS2IAPK4ILA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.931037770 (592h45m27.802120805s) [noise=true , address=145.79.227.14:4839 , piece_id=BWUAORX5JM4B7R32R32GTSMFSDC5RUM7YDDV5Z67BRLXKBCXIW4A, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.931205659 (592h45m27.802288694s) [noise=true , address=145.79.227.22:4509 , piece_id=CPVKNKKLDRQSHRWBZPW5GNC5N3ZQYIBQDUMJMOSPMNDW3KFZQQUQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:27.947385048 (592h45m27.818468083s) [noise=true , address=145.79.227.29:4943 , piece_id=U4CSBL3CA2IJHPKNFX5VRG2NMCNZ2PV4MY6OC44E5YXSWS6CKPUA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.000304415 (592h45m27.87138747s) [noise=true , address=145.79.227.8:4331 , piece_id=JQE2P5P322EACXUGWNHASSV6TZMIP5RWMOZMKR7MIMHBR43WWESQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.014732849 (592h45m27.885815894s) [noise=true , address=145.79.227.41:4087 , piece_id=KPOMWQGIFBR5VF2BQTKMG7XZAI5RZQGNM7KQ2WDLZILPST627WTQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.171983078 (592h45m28.043066113s) [noise=true , address=77.90.31.18:4805 , piece_id=SDTNLVREX26UFYEWQUDV6EX4YSNZJISGSQFS7ZUYFBKVGRBXSRZQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.213841073 (592h45m28.084924128s) [noise=true , address=45.91.116.13:4154 , piece_id=R3NPUQ5PXI4L5Q2YR2O7DMG7ZVJZ4GUCYAUQ5BCL7MJ6YJJ44O7A, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.222817134 (592h45m28.093900179s) [noise=true , address=45.91.116.12:4225 , piece_id=4RJ2HL5VOOVHDWFUQOWRNSND7EJ2D2MIG53R6X5SV4HBYNE2ORBQ, seg_pos={0 0}, upload=519659]: Done uploading piece. err: <nil> Dec 30 08:14:28.222843214 (592h45m28.093926249s) [seg_pos={0 0}, upload=519659]: Segment reached optimal threshold of 13 pieces. Dec 30 08:14:28.223173752 (592h45m28.094256798s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 30 08:14:28.223182590 (592h45m28.094265635s) [noise=true , address=77.90.31.10:4519 , piece_id=IKXTZGKGYP34QWCUCSKS6ERCIQYDLUSRZXY75NKWOJ22FB4YK23Q, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223546122 (592h45m28.094629167s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42780->77.90.31.13:4629: use of closed network connection Dec 30 08:14:28.223557032 (592h45m28.094640077s) [noise=true , address=77.90.31.13:4629 , piece_id=SCW4MYN4BVZFNAGMN4IFXBRLCWN3BDZ2AXBNUBW454JSJIQM2HHA, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223746033 (592h45m28.094829068s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 08:14:28.223752645 (592h45m28.09483568s) [noise=true , address=45.91.116.4:4940 , piece_id=JSCHVYGNXSSDO6PQRXC2D7UOSZL5VLD7FV5LZHQWD43CVHY4L7FA, seg_pos={0 0}, upload=519659]: Long tail cancellation due to optimal threshold Dec 30 08:14:28.223768846 (592h45m28.094851881s) [seg_pos={0 0}, upload=519659]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:14:28.223839430 (592h45m28.094922465s) [upload=519659]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:14:28.223845112 (592h45m28.094928147s) [upload=519659]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:14:28.507406377 (592h45m28.378489422s) [upload=519659]: Done uploading segments... Dec 30 08:14:28.507431966 (592h45m28.378515011s) [upload=519659]: Upload finished. err: <nil> Duration: 30.003521294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X................X...........] duration: 32.669291105s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:14:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 32.669307055s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X................] duration: 3.6077754s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:14:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.607813334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X................X..........] duration: 26.385496502s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:14:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 26.385527841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.XX..X....XX.XX..................X...X..X..X..XX.X.X...XX.X] duration: 30.00268788s trace: https://jaeger.dev.storj.io/trace/5cc7d8b6bf151295 FAIL (2025-12-30 08:14:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: XA65BNR2AM3AL2HMVZ3VSL6TYFJ3UTJ6QIAJY7FN5Z7UXSBXXELA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: UKEAICW7YLJLZD72HNKP6HZ4GPN2JVBJ5XREHU6DYNXBYFPS374A) context deadline exceeded ======LOGS====== Dec 30 08:13:43.179582799 (592h44m43.050665844s) [upload=519650]: Starting upload Dec 30 08:13:43.179661438 (592h44m43.050744513s) [upload=519650]: Done starting upload Dec 30 08:13:43.180012497 (592h44m43.051095532s) [upload=519650]: Uploading segments... Dec 30 08:13:43.180122256 (592h44m43.051205331s) [upload=519650]: Got next segment. Inline: false Dec 30 08:13:43.180205605 (592h44m43.05128865s) [upload=519650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:13:43.180212378 (592h44m43.051295413s) [upload=519650]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:13:43.430573033 (592h44m43.301656068s) [seg_pos={0 0}, upload=519650]: Begin upload segment... Dec 30 08:13:43.430635643 (592h44m43.301718688s) [seg_pos={0 0}, upload=519650]: Done begin upload segment. Dec 30 08:13:43.430690347 (592h44m43.301773392s) [noise=true , address=45.91.116.8:4048 , piece_id=SC4N3YUTG5YPZN25K5FCOHMQEHNDVG7ZJIXM2SESMSXJSHTLBMOQ, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.430721136 (592h44m43.301804171s) [noise=true , address=145.79.227.39:4013 , piece_id=IZ3LDOMKYV54TLZVZU3QIZ6S56ZS3W56Z35CFR43JPRVEZAO44FQ, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.431385982 (592h44m43.302469027s) [noise=true , address=145.79.227.3:4150 , piece_id=5YX24OX6I4C23BHV754VRFVFHPOL3ONE3KS7LNANJA6BUIT7SRRQ, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.431789019 (592h44m43.302872064s) [noise=true , address=145.79.227.5:4223 , piece_id=OAZOYFQS2RVQMO5YCD3MVY32QY5TQ4F7FTY6K3RZFTVEIZRUVD3Q, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.431956648 (592h44m43.303039683s) [noise=true , address=145.79.227.51:4409 , piece_id=GI6OEK72XMBS56XYDD42VLYVIBBXKUVD7NREKUS5EX7OTE7PCHMQ, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.432504552 (592h44m43.303587597s) [noise=true , address=145.79.227.21:4475 , piece_id=4CIYZ6RHJZCIVLNEZ6XEH5TQNTKH2LPMRVXHDOTICKLDKV5AB5UQ, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.432528588 (592h44m43.303611633s) [noise=true , address=145.79.227.17:4972 , piece_id=EBDURVAZTYX4KBCVKKVZBSU7JR6E2NM47BY5K6GF42W2J4NC6GSA, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.432984636 (592h44m43.304067681s) [noise=true , address=145.79.227.36:4518 , piece_id=RJNM6YM3GAOPLTH4HWYSFZC6XITMQ7CNFAC4JJ56IAZC7ATOKQGQ, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.433034130 (592h44m43.304117165s) [noise=true , address=145.79.227.23:4547 , piece_id=MSZTH66D67DLHDTBD4ZIA5UOZIM5LEALOFPP2WCTIINY2YIZPJWA, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.433425004 (592h44m43.304508039s) [noise=true , address=45.91.116.5:4299 , piece_id=XA65BNR2AM3AL2HMVZ3VSL6TYFJ3UTJ6QIAJY7FN5Z7UXSBXXELA, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.433960044 (592h44m43.305043079s) [noise=true , address=45.91.116.12:4227 , piece_id=BT4CEQVKTJDYESRIBCUBLIFETPRCQTGNI5GU45G5WDFBFSVNSPIA, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.434394681 (592h44m43.305477716s) [noise=true , address=77.90.31.12:4589 , piece_id=63Y6FLFI2VPPQYVWYHTIWQ3Z2BII73JAQVLNYKBPZ5VVTL7A6JIQ, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.434898150 (592h44m43.305981195s) [noise=true , address=77.90.31.19:4841 , piece_id=UKEAICW7YLJLZD72HNKP6HZ4GPN2JVBJ5XREHU6DYNXBYFPS374A, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.435523250 (592h44m43.306606295s) [noise=true , address=145.79.227.40:4051 , piece_id=EJ2AEXQS33X3HXL5V6WLYXLYWYWRACDOU2KHKIQXYOJIHOV3HN6A, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.436777248 (592h44m43.307860283s) [noise=true , address=145.79.227.42:4122 , piece_id=VZBFYD7SXNHZTYLNL7BR3W3EB42UPWQP5GEYTX2NOGP3OYNN7OOA, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:43.437774546 (592h44m43.308857582s) [noise=true , address=77.90.31.13:4627 , piece_id=H5PUIWJLWLTTWX47WRTBKZDAK2PPA7RQLGPTWS57GLJCGUHOFEMA, seg_pos={0 0}, upload=519650]: Uploading piece... Dec 30 08:13:45.133008101 (592h44m45.004091146s) [upload=519650]: Next returned nil segment Dec 30 08:13:45.133021326 (592h44m45.004104361s) [upload=519650]: Waiting for error group managing segments... Dec 30 08:13:45.416036381 (592h44m45.287119427s) [noise=true , address=145.79.227.39:4013 , piece_id=IZ3LDOMKYV54TLZVZU3QIZ6S56ZS3W56Z35CFR43JPRVEZAO44FQ, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.416221445 (592h44m45.28730449s) [noise=true , address=145.79.227.51:4409 , piece_id=GI6OEK72XMBS56XYDD42VLYVIBBXKUVD7NREKUS5EX7OTE7PCHMQ, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.422462238 (592h44m45.293545273s) [noise=true , address=145.79.227.3:4150 , piece_id=5YX24OX6I4C23BHV754VRFVFHPOL3ONE3KS7LNANJA6BUIT7SRRQ, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.425968965 (592h44m45.297052s) [noise=true , address=145.79.227.21:4475 , piece_id=4CIYZ6RHJZCIVLNEZ6XEH5TQNTKH2LPMRVXHDOTICKLDKV5AB5UQ, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.429985403 (592h44m45.301068438s) [noise=true , address=145.79.227.17:4972 , piece_id=EBDURVAZTYX4KBCVKKVZBSU7JR6E2NM47BY5K6GF42W2J4NC6GSA, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.442595254 (592h44m45.313678289s) [noise=true , address=145.79.227.5:4223 , piece_id=OAZOYFQS2RVQMO5YCD3MVY32QY5TQ4F7FTY6K3RZFTVEIZRUVD3Q, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.445408571 (592h44m45.316491617s) [noise=true , address=145.79.227.36:4518 , piece_id=RJNM6YM3GAOPLTH4HWYSFZC6XITMQ7CNFAC4JJ56IAZC7ATOKQGQ, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.508296800 (592h44m45.379379835s) [noise=true , address=145.79.227.42:4122 , piece_id=VZBFYD7SXNHZTYLNL7BR3W3EB42UPWQP5GEYTX2NOGP3OYNN7OOA, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.509305982 (592h44m45.380389027s) [noise=true , address=145.79.227.23:4547 , piece_id=MSZTH66D67DLHDTBD4ZIA5UOZIM5LEALOFPP2WCTIINY2YIZPJWA, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.689376555 (592h44m45.56045961s) [noise=true , address=77.90.31.13:4627 , piece_id=H5PUIWJLWLTTWX47WRTBKZDAK2PPA7RQLGPTWS57GLJCGUHOFEMA, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.711267529 (592h44m45.582350574s) [noise=true , address=77.90.31.19:4841 , piece_id=UKEAICW7YLJLZD72HNKP6HZ4GPN2JVBJ5XREHU6DYNXBYFPS374A, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.713300600 (592h44m45.584383635s) [noise=true , address=45.91.116.5:4299 , piece_id=XA65BNR2AM3AL2HMVZ3VSL6TYFJ3UTJ6QIAJY7FN5Z7UXSBXXELA, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.813687773 (592h44m45.684770818s) [noise=true , address=45.91.116.12:4227 , piece_id=BT4CEQVKTJDYESRIBCUBLIFETPRCQTGNI5GU45G5WDFBFSVNSPIA, seg_pos={0 0}, upload=519650]: Done uploading piece. err: <nil> Dec 30 08:13:45.813711248 (592h44m45.684794283s) [seg_pos={0 0}, upload=519650]: Segment reached optimal threshold of 13 pieces. Dec 30 08:13:45.814114836 (592h44m45.685197891s) [noise=true , address=45.91.116.8:4048 , piece_id=SC4N3YUTG5YPZN25K5FCOHMQEHNDVG7ZJIXM2SESMSXJSHTLBMOQ, seg_pos={0 0}, upload=519650]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 08:13:45.814126299 (592h44m45.685209354s) [noise=true , address=45.91.116.8:4048 , piece_id=SC4N3YUTG5YPZN25K5FCOHMQEHNDVG7ZJIXM2SESMSXJSHTLBMOQ, seg_pos={0 0}, upload=519650]: Long tail cancellation due to optimal threshold Dec 30 08:13:45.814153289 (592h44m45.685236334s) [noise=true , address=145.79.227.40:4051 , piece_id=EJ2AEXQS33X3HXL5V6WLYXLYWYWRACDOU2KHKIQXYOJIHOV3HN6A, seg_pos={0 0}, upload=519650]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 08:13:45.814161385 (592h44m45.68524442s) [noise=true , address=145.79.227.40:4051 , piece_id=EJ2AEXQS33X3HXL5V6WLYXLYWYWRACDOU2KHKIQXYOJIHOV3HN6A, seg_pos={0 0}, upload=519650]: Long tail cancellation due to optimal threshold Dec 30 08:13:45.814336779 (592h44m45.685419814s) [noise=true , address=77.90.31.12:4589 , piece_id=63Y6FLFI2VPPQYVWYHTIWQ3Z2BII73JAQVLNYKBPZ5VVTL7A6JIQ, seg_pos={0 0}, upload=519650]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: read tcp 188.166.225.170:50258->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 08:13:45.814343282 (592h44m45.685426327s) [noise=true , address=77.90.31.12:4589 , piece_id=63Y6FLFI2VPPQYVWYHTIWQ3Z2BII73JAQVLNYKBPZ5VVTL7A6JIQ, seg_pos={0 0}, upload=519650]: Long tail cancellation due to optimal threshold Dec 30 08:13:45.814399147 (592h44m45.685482182s) [seg_pos={0 0}, upload=519650]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:13:45.814463530 (592h44m45.685546565s) [upload=519650]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:13:45.814468489 (592h44m45.685551524s) [upload=519650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:13:46.116992784 (592h44m45.988075819s) [upload=519650]: Done uploading segments... Dec 30 08:13:46.117012813 (592h44m45.988095848s) [upload=519650]: Upload finished. err: <nil> Duration: 30.002804733s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X...............] duration: 3.476486274s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:13:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.476508597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X................X.........] duration: 16.619789364s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:13:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 16.619807199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X................X........] duration: 42.021946307s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:13:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 42.021959301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.XX..X....XX.XX..................X...X..X..X..XX.X.X...XX.] duration: 7.381062426s trace: https://jaeger.dev.storj.io/trace/59c2c5eb3d70dd85 FAIL (2025-12-30 08:13:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JABMWLXCXWOYWQVP3X6H4Y3SCNCILYRNRMPUQOIEJQYRESAOOGDQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KP5D2XO4657YMS3GRIEG4THHEN4BDR7C5NBP7FVLECQR5U24AZFQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: A6IE2PBM7FIEDLPNL5L3QWQA3MPRXAYDU6Q2CNAH7XWDYXDJGZIQ) context deadline exceeded ======LOGS====== ObjectBegin Dec 30 08:12:02.128927708 (592h43m2.000010753s) [upload=519624]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:12:02.376750130 (592h43m2.247833175s) [seg_pos={0 0}, upload=519624]: Begin upload segment... Dec 30 08:12:02.376801318 (592h43m2.247884363s) [seg_pos={0 0}, upload=519624]: Done begin upload segment. Dec 30 08:12:02.376837577 (592h43m2.247920622s) [noise=true , address=145.79.227.14:4837 , piece_id=6QOGKI6IUYM5YNKL4SGJOOHNL4XWPNTX7ZMUSE2GXSURS3L6ZSJA, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.376847155 (592h43m2.2479302s) [noise=true , address=145.79.227.18:4006 , piece_id=JNMCSBZCJUSIVBSEO5L4V6FNIGGZSQG6UE4QZX3C5YNSTU4RX2JQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.376909133 (592h43m2.247992168s) [noise=true , address=45.91.116.13:4155 , piece_id=A6IE2PBM7FIEDLPNL5L3QWQA3MPRXAYDU6Q2CNAH7XWDYXDJGZIQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.377587435 (592h43m2.24867048s) [noise=true , address=145.79.227.8:4329 , piece_id=MYO6DMOBJ2LLQNQ36HXA2Z3AOQWSX7SWUMSG7JGTTXN4CY6EIWSA, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.378084241 (592h43m2.249167276s) [noise=true , address=145.79.227.39:4015 , piece_id=QHVDDE6FTKXVYHI45W37HZ3BZYLZDFVO7RO35MT2FCNAAQ3K5TZA, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.378546211 (592h43m2.249629256s) [noise=true , address=145.79.227.2:4114 , piece_id=SNZAJV7PEOTLXXVKFU53YXFAZFIX4ZP65P57HAHT4Q2WYQH5427Q, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.379102891 (592h43m2.250185936s) [noise=true , address=77.90.31.19:4842 , piece_id=URC6FSS2TPR7FS44RXALRXU5CEWCVR6A4ARZ5K27XBANSHEIOURQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.379632629 (592h43m2.250715664s) [noise=true , address=145.79.227.26:4653 , piece_id=VSQ7CCFXZ7CO6Z6RPQOM32JLIMWGVVRUPCJYHXPNM6R7SRTLXXYQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.380049834 (592h43m2.251132879s) [noise=true , address=145.79.227.16:4912 , piece_id=2L5CDWYPEJDADUPBP5RTV4MJQKXLDXIANMOV3H6IJUTY2YYJF3KQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.380470895 (592h43m2.25155394s) [noise=true , address=145.79.227.42:4122 , piece_id=SQDE2SS57DS6KRDQVUXHV32RXMPLFDSP6MG256N2Z2YN5AMIW2IA, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.381894256 (592h43m2.252977301s) [noise=true , address=77.90.31.16:4736 , piece_id=JABMWLXCXWOYWQVP3X6H4Y3SCNCILYRNRMPUQOIEJQYRESAOOGDQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.383643146 (592h43m2.254726181s) [noise=true , address=145.79.227.3:4151 , piece_id=YYQH5CSOLJ3F2EEA7QAOKVD5HNOH3XMUEXH3OWRQNVBKLYB5FQZQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.385354265 (592h43m2.2564373s) [noise=true , address=45.91.116.8:4046 , piece_id=HIAZNTVNMIPRXJKY4CM2ML7VR3M5NBLSHJCVA4S6XPKQRCBD2RAA, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.387250705 (592h43m2.258333751s) [noise=true , address=145.79.227.43:4158 , piece_id=KDAH6EF4AA6NANLRWLORAS3YFEUF56BZMTDMP4UWTGONRIHA7TAQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.388763877 (592h43m2.259846922s) [noise=true , address=45.91.116.9:4262 , piece_id=KP5D2XO4657YMS3GRIEG4THHEN4BDR7C5NBP7FVLECQR5U24AZFQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.391483406 (592h43m2.262566451s) [noise=true , address=145.79.227.11:4730 , piece_id=2SWAWOQWP7R6W4BYGM3HPGKY27Z6UQ4VKSN3FACYMBL5MZWU2M6Q, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:03.991256208 (592h43m3.862339243s) [upload=519624]: Next returned nil segment Dec 30 08:12:03.991269984 (592h43m3.862353029s) [upload=519624]: Waiting for error group managing segments... Dec 30 08:12:04.372519004 (592h43m4.243602069s) [noise=true , address=145.79.227.8:4329 , piece_id=MYO6DMOBJ2LLQNQ36HXA2Z3AOQWSX7SWUMSG7JGTTXN4CY6EIWSA, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.372679879 (592h43m4.243762915s) [noise=true , address=145.79.227.16:4912 , piece_id=2L5CDWYPEJDADUPBP5RTV4MJQKXLDXIANMOV3H6IJUTY2YYJF3KQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.373273570 (592h43m4.244356625s) [noise=true , address=145.79.227.2:4114 , piece_id=SNZAJV7PEOTLXXVKFU53YXFAZFIX4ZP65P57HAHT4Q2WYQH5427Q, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.386843229 (592h43m4.257926274s) [noise=true , address=145.79.227.42:4122 , piece_id=SQDE2SS57DS6KRDQVUXHV32RXMPLFDSP6MG256N2Z2YN5AMIW2IA, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.389379337 (592h43m4.260462372s) [noise=true , address=145.79.227.26:4653 , piece_id=VSQ7CCFXZ7CO6Z6RPQOM32JLIMWGVVRUPCJYHXPNM6R7SRTLXXYQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.389463608 (592h43m4.260546654s) [noise=true , address=145.79.227.14:4837 , piece_id=6QOGKI6IUYM5YNKL4SGJOOHNL4XWPNTX7ZMUSE2GXSURS3L6ZSJA, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.404765044 (592h43m4.27584809s) [noise=true , address=145.79.227.43:4158 , piece_id=KDAH6EF4AA6NANLRWLORAS3YFEUF56BZMTDMP4UWTGONRIHA7TAQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.554790507 (592h43m4.425873552s) [noise=true , address=77.90.31.19:4842 , piece_id=URC6FSS2TPR7FS44RXALRXU5CEWCVR6A4ARZ5K27XBANSHEIOURQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.865660232 (592h43m4.736743278s) [noise=true , address=77.90.31.16:4736 , piece_id=JABMWLXCXWOYWQVP3X6H4Y3SCNCILYRNRMPUQOIEJQYRESAOOGDQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:05.049118616 (592h43m4.920201661s) [noise=true , address=45.91.116.9:4262 , piece_id=KP5D2XO4657YMS3GRIEG4THHEN4BDR7C5NBP7FVLECQR5U24AZFQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:05.067954954 (592h43m4.939037999s) [noise=true , address=45.91.116.13:4155 , piece_id=A6IE2PBM7FIEDLPNL5L3QWQA3MPRXAYDU6Q2CNAH7XWDYXDJGZIQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:05.083760520 (592h43m4.954843565s) [noise=true , address=45.91.116.8:4046 , piece_id=HIAZNTVNMIPRXJKY4CM2ML7VR3M5NBLSHJCVA4S6XPKQRCBD2RAA, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:26.172097400 (592h43m26.043180445s) [noise=true , address=145.79.227.11:4730 , piece_id=2SWAWOQWP7R6W4BYGM3HPGKY27Z6UQ4VKSN3FACYMBL5MZWU2M6Q, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:26.172123920 (592h43m26.043206965s) [seg_pos={0 0}, upload=519624]: Segment reached optimal threshold of 13 pieces. Dec 30 08:12:26.172350060 (592h43m26.043433105s) [noise=true , address=145.79.227.18:4006 , piece_id=JNMCSBZCJUSIVBSEO5L4V6FNIGGZSQG6UE4QZX3C5YNSTU4RX2JQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 188.166.225.170:36634->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:36634->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36634->145.79.227.18:4006: use of closed network connection Dec 30 08:12:26.172358256 (592h43m26.043441301s) [noise=true , address=145.79.227.18:4006 , piece_id=JNMCSBZCJUSIVBSEO5L4V6FNIGGZSQG6UE4QZX3C5YNSTU4RX2JQ, seg_pos={0 0}, upload=519624]: Long tail cancellation due to optimal threshold Dec 30 08:12:26.172649461 (592h43m26.043732496s) [noise=true , address=145.79.227.3:4151 , piece_id=YYQH5CSOLJ3F2EEA7QAOKVD5HNOH3XMUEXH3OWRQNVBKLYB5FQZQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:48392->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:48392->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48392->145.79.227.3:4151: use of closed network connection Dec 30 08:12:26.172659850 (592h43m26.043742895s) [noise=true , address=145.79.227.3:4151 , piece_id=YYQH5CSOLJ3F2EEA7QAOKVD5HNOH3XMUEXH3OWRQNVBKLYB5FQZQ, seg_pos={0 0}, upload=519624]: Long tail cancellation due to optimal threshold Dec 30 08:12:26.172776021 (592h43m26.043859066s) [noise=true , address=145.79.227.39:4015 , piece_id=QHVDDE6FTKXVYHI45W37HZ3BZYLZDFVO7RO35MT2FCNAAQ3K5TZA, seg_pos={0 0}, upload=519624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:45798->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45798->145.79.227.39:4015: use of closed network connection Dec 30 08:12:26.172782945 (592h43m26.04386599s) [noise=true , address=145.79.227.39:4015 , piece_id=QHVDDE6FTKXVYHI45W37HZ3BZYLZDFVO7RO35MT2FCNAAQ3K5TZA, seg_pos={0 0}, upload=519624]: Long tail cancellation due to optimal threshold Dec 30 08:12:26.172804355 (592h43m26.0438874s) [seg_pos={0 0}, upload=519624]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:12:26.172893546 (592h43m26.043976591s) [upload=519624]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:12:26.172899657 (592h43m26.043982702s) [upload=519624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:12:26.474059092 (592h43m26.345142127s) [upload=519624]: Done uploading segments... Dec 30 08:12:26.474077868 (592h43m26.345160903s) [upload=519624]: Upload finished. err: <nil> Duration:* 7.38109089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X..............] duration: 3.52308785s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:12:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.523107216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X................X.......] duration: 36.748881387s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:12:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 36.748910181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.XX..X....XX.XX..................X...X..X..X..XX.X.X...XX] duration: 30.003181321s trace: https://jaeger.dev.storj.io/trace/59c2c5eb3d70dd85 FAIL (2025-12-30 08:12:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JABMWLXCXWOYWQVP3X6H4Y3SCNCILYRNRMPUQOIEJQYRESAOOGDQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KP5D2XO4657YMS3GRIEG4THHEN4BDR7C5NBP7FVLECQR5U24AZFQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: A6IE2PBM7FIEDLPNL5L3QWQA3MPRXAYDU6Q2CNAH7XWDYXDJGZIQ) context deadline exceeded ======LOGS====== ObjectBegin Dec 30 08:12:02.128927708 (592h43m2.000010753s) [upload=519624]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:12:02.376750130 (592h43m2.247833175s) [seg_pos={0 0}, upload=519624]: Begin upload segment... Dec 30 08:12:02.376801318 (592h43m2.247884363s) [seg_pos={0 0}, upload=519624]: Done begin upload segment. Dec 30 08:12:02.376837577 (592h43m2.247920622s) [noise=true , address=145.79.227.14:4837 , piece_id=6QOGKI6IUYM5YNKL4SGJOOHNL4XWPNTX7ZMUSE2GXSURS3L6ZSJA, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.376847155 (592h43m2.2479302s) [noise=true , address=145.79.227.18:4006 , piece_id=JNMCSBZCJUSIVBSEO5L4V6FNIGGZSQG6UE4QZX3C5YNSTU4RX2JQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.376909133 (592h43m2.247992168s) [noise=true , address=45.91.116.13:4155 , piece_id=A6IE2PBM7FIEDLPNL5L3QWQA3MPRXAYDU6Q2CNAH7XWDYXDJGZIQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.377587435 (592h43m2.24867048s) [noise=true , address=145.79.227.8:4329 , piece_id=MYO6DMOBJ2LLQNQ36HXA2Z3AOQWSX7SWUMSG7JGTTXN4CY6EIWSA, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.378084241 (592h43m2.249167276s) [noise=true , address=145.79.227.39:4015 , piece_id=QHVDDE6FTKXVYHI45W37HZ3BZYLZDFVO7RO35MT2FCNAAQ3K5TZA, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.378546211 (592h43m2.249629256s) [noise=true , address=145.79.227.2:4114 , piece_id=SNZAJV7PEOTLXXVKFU53YXFAZFIX4ZP65P57HAHT4Q2WYQH5427Q, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.379102891 (592h43m2.250185936s) [noise=true , address=77.90.31.19:4842 , piece_id=URC6FSS2TPR7FS44RXALRXU5CEWCVR6A4ARZ5K27XBANSHEIOURQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.379632629 (592h43m2.250715664s) [noise=true , address=145.79.227.26:4653 , piece_id=VSQ7CCFXZ7CO6Z6RPQOM32JLIMWGVVRUPCJYHXPNM6R7SRTLXXYQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.380049834 (592h43m2.251132879s) [noise=true , address=145.79.227.16:4912 , piece_id=2L5CDWYPEJDADUPBP5RTV4MJQKXLDXIANMOV3H6IJUTY2YYJF3KQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.380470895 (592h43m2.25155394s) [noise=true , address=145.79.227.42:4122 , piece_id=SQDE2SS57DS6KRDQVUXHV32RXMPLFDSP6MG256N2Z2YN5AMIW2IA, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.381894256 (592h43m2.252977301s) [noise=true , address=77.90.31.16:4736 , piece_id=JABMWLXCXWOYWQVP3X6H4Y3SCNCILYRNRMPUQOIEJQYRESAOOGDQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.383643146 (592h43m2.254726181s) [noise=true , address=145.79.227.3:4151 , piece_id=YYQH5CSOLJ3F2EEA7QAOKVD5HNOH3XMUEXH3OWRQNVBKLYB5FQZQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.385354265 (592h43m2.2564373s) [noise=true , address=45.91.116.8:4046 , piece_id=HIAZNTVNMIPRXJKY4CM2ML7VR3M5NBLSHJCVA4S6XPKQRCBD2RAA, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.387250705 (592h43m2.258333751s) [noise=true , address=145.79.227.43:4158 , piece_id=KDAH6EF4AA6NANLRWLORAS3YFEUF56BZMTDMP4UWTGONRIHA7TAQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.388763877 (592h43m2.259846922s) [noise=true , address=45.91.116.9:4262 , piece_id=KP5D2XO4657YMS3GRIEG4THHEN4BDR7C5NBP7FVLECQR5U24AZFQ, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:02.391483406 (592h43m2.262566451s) [noise=true , address=145.79.227.11:4730 , piece_id=2SWAWOQWP7R6W4BYGM3HPGKY27Z6UQ4VKSN3FACYMBL5MZWU2M6Q, seg_pos={0 0}, upload=519624]: Uploading piece... Dec 30 08:12:03.991256208 (592h43m3.862339243s) [upload=519624]: Next returned nil segment Dec 30 08:12:03.991269984 (592h43m3.862353029s) [upload=519624]: Waiting for error group managing segments... Dec 30 08:12:04.372519004 (592h43m4.243602069s) [noise=true , address=145.79.227.8:4329 , piece_id=MYO6DMOBJ2LLQNQ36HXA2Z3AOQWSX7SWUMSG7JGTTXN4CY6EIWSA, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.372679879 (592h43m4.243762915s) [noise=true , address=145.79.227.16:4912 , piece_id=2L5CDWYPEJDADUPBP5RTV4MJQKXLDXIANMOV3H6IJUTY2YYJF3KQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.373273570 (592h43m4.244356625s) [noise=true , address=145.79.227.2:4114 , piece_id=SNZAJV7PEOTLXXVKFU53YXFAZFIX4ZP65P57HAHT4Q2WYQH5427Q, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.386843229 (592h43m4.257926274s) [noise=true , address=145.79.227.42:4122 , piece_id=SQDE2SS57DS6KRDQVUXHV32RXMPLFDSP6MG256N2Z2YN5AMIW2IA, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.389379337 (592h43m4.260462372s) [noise=true , address=145.79.227.26:4653 , piece_id=VSQ7CCFXZ7CO6Z6RPQOM32JLIMWGVVRUPCJYHXPNM6R7SRTLXXYQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.389463608 (592h43m4.260546654s) [noise=true , address=145.79.227.14:4837 , piece_id=6QOGKI6IUYM5YNKL4SGJOOHNL4XWPNTX7ZMUSE2GXSURS3L6ZSJA, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.404765044 (592h43m4.27584809s) [noise=true , address=145.79.227.43:4158 , piece_id=KDAH6EF4AA6NANLRWLORAS3YFEUF56BZMTDMP4UWTGONRIHA7TAQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.554790507 (592h43m4.425873552s) [noise=true , address=77.90.31.19:4842 , piece_id=URC6FSS2TPR7FS44RXALRXU5CEWCVR6A4ARZ5K27XBANSHEIOURQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:04.865660232 (592h43m4.736743278s) [noise=true , address=77.90.31.16:4736 , piece_id=JABMWLXCXWOYWQVP3X6H4Y3SCNCILYRNRMPUQOIEJQYRESAOOGDQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:05.049118616 (592h43m4.920201661s) [noise=true , address=45.91.116.9:4262 , piece_id=KP5D2XO4657YMS3GRIEG4THHEN4BDR7C5NBP7FVLECQR5U24AZFQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:05.067954954 (592h43m4.939037999s) [noise=true , address=45.91.116.13:4155 , piece_id=A6IE2PBM7FIEDLPNL5L3QWQA3MPRXAYDU6Q2CNAH7XWDYXDJGZIQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:05.083760520 (592h43m4.954843565s) [noise=true , address=45.91.116.8:4046 , piece_id=HIAZNTVNMIPRXJKY4CM2ML7VR3M5NBLSHJCVA4S6XPKQRCBD2RAA, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:26.172097400 (592h43m26.043180445s) [noise=true , address=145.79.227.11:4730 , piece_id=2SWAWOQWP7R6W4BYGM3HPGKY27Z6UQ4VKSN3FACYMBL5MZWU2M6Q, seg_pos={0 0}, upload=519624]: Done uploading piece. err: <nil> Dec 30 08:12:26.172123920 (592h43m26.043206965s) [seg_pos={0 0}, upload=519624]: Segment reached optimal threshold of 13 pieces. Dec 30 08:12:26.172350060 (592h43m26.043433105s) [noise=true , address=145.79.227.18:4006 , piece_id=JNMCSBZCJUSIVBSEO5L4V6FNIGGZSQG6UE4QZX3C5YNSTU4RX2JQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 188.166.225.170:36634->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:36634->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36634->145.79.227.18:4006: use of closed network connection Dec 30 08:12:26.172358256 (592h43m26.043441301s) [noise=true , address=145.79.227.18:4006 , piece_id=JNMCSBZCJUSIVBSEO5L4V6FNIGGZSQG6UE4QZX3C5YNSTU4RX2JQ, seg_pos={0 0}, upload=519624]: Long tail cancellation due to optimal threshold Dec 30 08:12:26.172649461 (592h43m26.043732496s) [noise=true , address=145.79.227.3:4151 , piece_id=YYQH5CSOLJ3F2EEA7QAOKVD5HNOH3XMUEXH3OWRQNVBKLYB5FQZQ, seg_pos={0 0}, upload=519624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:48392->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:48392->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48392->145.79.227.3:4151: use of closed network connection Dec 30 08:12:26.172659850 (592h43m26.043742895s) [noise=true , address=145.79.227.3:4151 , piece_id=YYQH5CSOLJ3F2EEA7QAOKVD5HNOH3XMUEXH3OWRQNVBKLYB5FQZQ, seg_pos={0 0}, upload=519624]: Long tail cancellation due to optimal threshold Dec 30 08:12:26.172776021 (592h43m26.043859066s) [noise=true , address=145.79.227.39:4015 , piece_id=QHVDDE6FTKXVYHI45W37HZ3BZYLZDFVO7RO35MT2FCNAAQ3K5TZA, seg_pos={0 0}, upload=519624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:45798->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45798->145.79.227.39:4015: use of closed network connection Dec 30 08:12:26.172782945 (592h43m26.04386599s) [noise=true , address=145.79.227.39:4015 , piece_id=QHVDDE6FTKXVYHI45W37HZ3BZYLZDFVO7RO35MT2FCNAAQ3K5TZA, seg_pos={0 0}, upload=519624]: Long tail cancellation due to optimal threshold Dec 30 08:12:26.172804355 (592h43m26.0438874s) [seg_pos={0 0}, upload=519624]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:12:26.172893546 (592h43m26.043976591s) [upload=519624]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:12:26.172899657 (592h43m26.043982702s) [upload=519624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:12:26.474059092 (592h43m26.345142127s) [upload=519624]: Done uploading segments... Dec 30 08:12:26.474077868 (592h43m26.345160903s) [upload=519624]: Upload finished. err: <nil> Duration:* 30.003200156s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.XX..X....XX.XX..................X...X..X..X..XX.X.X...X] duration: 30.002300375s trace: https://jaeger.dev.storj.io/trace/2bdcce7d2f801eb FAIL (2025-12-30 08:11:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 08:11:19.177027046 (592h42m19.048110081s) [upload=519616]: Starting upload Dec 30 08:11:19.177084776 (592h42m19.048167811s) [upload=519616]: Done starting upload Dec 30 08:11:19.177186209 (592h42m19.048269245s) [upload=519616]: Uploading segments... Dec 30 08:11:19.177295127 (592h42m19.048378172s) [upload=519616]: Got next segment. Inline: false Dec 30 08:11:19.177324974 (592h42m19.048408009s) [upload=519616]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:11:19.177330865 (592h42m19.0484139s) [upload=519616]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:11:19.419051951 (592h42m19.290134996s) [seg_pos={0 0}, upload=519616]: Begin upload segment... Dec 30 08:11:19.419099952 (592h42m19.290182987s) [seg_pos={0 0}, upload=519616]: Done begin upload segment. Dec 30 08:11:19.419163573 (592h42m19.290246608s) [noise=true , address=45.91.116.5:4299 , piece_id=32WIBBJK6W4IYOGUOD66LCWVN4TJCSG5KJCPF6GEIR6XP3IMV7PA, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.419411375 (592h42m19.29049441s) [noise=true , address=145.79.227.23:4545 , piece_id=VHWV4RCTYKWCJMUGFJGNR4TPEPLCGDXWBKSTVYI4VMSN4XVBDXYA, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.420131737 (592h42m19.291214772s) [noise=true , address=145.79.227.13:4802 , piece_id=64C22NOVAXI2I4U2RZHOAYM2UNVSIXFE4XBCNYHQCRKF4L5VVFGA, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.420573687 (592h42m19.291656733s) [noise=true , address=145.79.227.48:4303 , piece_id=AEV6U6VH27HW6EY7IZKO7S4KOF2JUGZ4RZBL2YGYRAX2QOQWRLBA, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.421545829 (592h42m19.292629154s) [noise=true , address=45.91.116.14:4191 , piece_id=3IKOUYCURVDJBHLI5CFFS24LIWWHFPLTU7BT2IKC2B567ETE3ZIA, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.422129290 (592h42m19.293212325s) [noise=true , address=145.79.227.14:4840 , piece_id=SQKQLTOOWRIGUHX7LDHIBHESVK4SO354F2IZIFNFUGGW6MEWWLKQ, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.422531435 (592h42m19.29361448s) [noise=true , address=145.79.227.36:4519 , piece_id=RL3IKI4BJUPUDUAKA74JR32RZPDSDLA2A3VG7G63TVU3ZW373IYQ, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.423905432 (592h42m19.294988467s) [noise=true , address=77.90.31.17:4770 , piece_id=I7Y26WQKKQXWZ6TTH2VZDJMZWKC2AI773PJC7QX2QQCJWCMGHCGA, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.424420814 (592h42m19.295503889s) [noise=true , address=77.90.31.11:4553 , piece_id=GW7NDC37AUV3HEKKAONZRA4RTLF67NM3N43C5XTWVPOIZ4ZP3OWA, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.425975734 (592h42m19.297058779s) [noise=true , address=145.79.227.28:4727 , piece_id=B4SMHAZUBPELLZ3QLJQUNCXY5Z5KT5XEKUK7L6YHOWD42FBA644Q, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.425998998 (592h42m19.297082043s) [noise=true , address=145.79.227.38:4979 , piece_id=TRTP2GPYL7C77DI46LSLO5Y6HV32A357A4OSX65A2TWJK6PQYILQ, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.428770656 (592h42m19.299853702s) [noise=true , address=145.79.227.16:4912 , piece_id=EUKD4ATEGKITEQ5IUJZUWRFPAUWMXMEI3N3QGA3A2YKEFGBHUSFA, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.430759574 (592h42m19.301842619s) [noise=true , address=77.90.31.15:4697 , piece_id=FKYZ237H5YN2I7W6JAWOOT6SJ23FA25AV23EUG5EQGRR3XHW3CUQ, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.431337093 (592h42m19.302420128s) [noise=true , address=145.79.227.39:4014 , piece_id=JBWRZEAWTJY4JD2AVXGPJFAA2UBHTMEEBRG6LZ52E7K3YQFJDLFA, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.435919759 (592h42m19.307002804s) [noise=true , address=145.79.227.32:4834 , piece_id=JLEBR462LRUIODAKOCGURWKQLBSZWEXCEABH2Q7FVECHLNCE6FKA, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:19.438748185 (592h42m19.30983123s) [noise=true , address=45.91.116.8:4047 , piece_id=KWS75SCK4LM3AYVTF5POJDZPW5TUOEYJ7TQMN5XO6ZX4BQDLS7DA, seg_pos={0 0}, upload=519616]: Uploading piece... Dec 30 08:11:21.143731875 (592h42m21.014814911s) [upload=519616]: Next returned nil segment Dec 30 08:11:21.143745101 (592h42m21.014828136s) [upload=519616]: Waiting for error group managing segments... Dec 30 08:11:21.416920942 (592h42m21.288003977s) [noise=true , address=145.79.227.16:4912 , piece_id=EUKD4ATEGKITEQ5IUJZUWRFPAUWMXMEI3N3QGA3A2YKEFGBHUSFA, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.425383014 (592h42m21.296466049s) [noise=true , address=145.79.227.14:4840 , piece_id=SQKQLTOOWRIGUHX7LDHIBHESVK4SO354F2IZIFNFUGGW6MEWWLKQ, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.430843552 (592h42m21.301926597s) [noise=true , address=145.79.227.38:4979 , piece_id=TRTP2GPYL7C77DI46LSLO5Y6HV32A357A4OSX65A2TWJK6PQYILQ, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.432462014 (592h42m21.303545059s) [noise=true , address=145.79.227.32:4834 , piece_id=JLEBR462LRUIODAKOCGURWKQLBSZWEXCEABH2Q7FVECHLNCE6FKA, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.438517264 (592h42m21.30960031s) [noise=true , address=145.79.227.36:4519 , piece_id=RL3IKI4BJUPUDUAKA74JR32RZPDSDLA2A3VG7G63TVU3ZW373IYQ, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.438736862 (592h42m21.309819897s) [noise=true , address=145.79.227.39:4014 , piece_id=JBWRZEAWTJY4JD2AVXGPJFAA2UBHTMEEBRG6LZ52E7K3YQFJDLFA, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.439187771 (592h42m21.310270806s) [noise=true , address=145.79.227.48:4303 , piece_id=AEV6U6VH27HW6EY7IZKO7S4KOF2JUGZ4RZBL2YGYRAX2QOQWRLBA, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.456661253 (592h42m21.327744288s) [noise=true , address=145.79.227.13:4802 , piece_id=64C22NOVAXI2I4U2RZHOAYM2UNVSIXFE4XBCNYHQCRKF4L5VVFGA, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.495521759 (592h42m21.366604794s) [noise=true , address=145.79.227.23:4545 , piece_id=VHWV4RCTYKWCJMUGFJGNR4TPEPLCGDXWBKSTVYI4VMSN4XVBDXYA, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.517983771 (592h42m21.389066806s) [noise=true , address=145.79.227.28:4727 , piece_id=B4SMHAZUBPELLZ3QLJQUNCXY5Z5KT5XEKUK7L6YHOWD42FBA644Q, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.607146000 (592h42m21.478229035s) [noise=true , address=77.90.31.17:4770 , piece_id=I7Y26WQKKQXWZ6TTH2VZDJMZWKC2AI773PJC7QX2QQCJWCMGHCGA, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.701979618 (592h42m21.573062663s) [noise=true , address=77.90.31.15:4697 , piece_id=FKYZ237H5YN2I7W6JAWOOT6SJ23FA25AV23EUG5EQGRR3XHW3CUQ, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.747540714 (592h42m21.618623749s) [noise=true , address=45.91.116.5:4299 , piece_id=32WIBBJK6W4IYOGUOD66LCWVN4TJCSG5KJCPF6GEIR6XP3IMV7PA, seg_pos={0 0}, upload=519616]: Done uploading piece. err: <nil> Dec 30 08:11:21.747558628 (592h42m21.618641663s) [seg_pos={0 0}, upload=519616]: Segment reached optimal threshold of 13 pieces. Dec 30 08:11:21.747733070 (592h42m21.618816115s) [noise=true , address=77.90.31.11:4553 , piece_id=GW7NDC37AUV3HEKKAONZRA4RTLF67NM3N43C5XTWVPOIZ4ZP3OWA, seg_pos={0 0}, upload=519616]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 08:11:21.747739252 (592h42m21.618822287s) [noise=true , address=77.90.31.11:4553 , piece_id=GW7NDC37AUV3HEKKAONZRA4RTLF67NM3N43C5XTWVPOIZ4ZP3OWA, seg_pos={0 0}, upload=519616]: Long tail cancellation due to optimal threshold Dec 30 08:11:21.748084850 (592h42m21.619167895s) [noise=true , address=45.91.116.8:4047 , piece_id=KWS75SCK4LM3AYVTF5POJDZPW5TUOEYJ7TQMN5XO6ZX4BQDLS7DA, seg_pos={0 0}, upload=519616]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 08:11:21.748106141 (592h42m21.619189186s) [noise=true , address=45.91.116.8:4047 , piece_id=KWS75SCK4LM3AYVTF5POJDZPW5TUOEYJ7TQMN5XO6ZX4BQDLS7DA, seg_pos={0 0}, upload=519616]: Long tail cancellation due to optimal threshold Dec 30 08:11:21.748181765 (592h42m21.6192648s) [noise=true , address=45.91.116.14:4191 , piece_id=3IKOUYCURVDJBHLI5CFFS24LIWWHFPLTU7BT2IKC2B567ETE3ZIA, seg_pos={0 0}, upload=519616]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 30 08:11:21.748195350 (592h42m21.619278385s) [noise=true , address=45.91.116.14:4191 , piece_id=3IKOUYCURVDJBHLI5CFFS24LIWWHFPLTU7BT2IKC2B567ETE3ZIA, seg_pos={0 0}, upload=519616]: Long tail cancellation due to optimal threshold Dec 30 08:11:21.748217212 (592h42m21.619300247s) [seg_pos={0 0}, upload=519616]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:11:21.748316050 (592h42m21.619399085s) [upload=519616]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:11:21.748321330 (592h42m21.619404365s) [upload=519616]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:11:22.029783789 (592h42m21.900866824s) [upload=519616]: Done uploading segments... Dec 30 08:11:22.029837983 (592h42m21.900921028s) [upload=519616]: Upload finished. err: <nil> Duration: 30.002331945s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [..X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X.............] duration: 3.356688537s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:11:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.356737731s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X................X......] duration: 28.06773766s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:11:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 28.067767386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X................X.....] duration: 35.607219285s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:10:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 35.607249304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X............] duration: 3.392996463s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:10:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39301573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.XX..X....XX.XX..................X...X..X..X..XX.X.X...] duration: 5.739981238s trace: https://jaeger.dev.storj.io/trace/3091684ebdfebfaf FAIL (2025-12-30 08:10:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ) context deadline exceeded ======LOGS====== egments... Dec 30 08:08:56.906787962 (592h39m56.777871007s) [upload=519577]: Got next segment. Inline: false Dec 30 08:08:56.906835392 (592h39m56.777918427s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:08:56.906849899 (592h39m56.777932945s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:08:57.151984305 (592h39m57.02306735s) [seg_pos={0 0}, upload=519577]: Begin upload segment... Dec 30 08:08:57.152045421 (592h39m57.023128466s) [seg_pos={0 0}, upload=519577]: Done begin upload segment. Dec 30 08:08:57.152088784 (592h39m57.023171829s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152103782 (592h39m57.023186817s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152948079 (592h39m57.024031114s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153318124 (592h39m57.024401159s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153610311 (592h39m57.024693346s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154041302 (592h39m57.025124337s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154116154 (592h39m57.025199199s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154499845 (592h39m57.02558289s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154851113 (592h39m57.025934158s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155328523 (592h39m57.026411558s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155630638 (592h39m57.026713673s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156234418 (592h39m57.027317463s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156721465 (592h39m57.02780451s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.158361338 (592h39m57.029444383s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.159690889 (592h39m57.030773924s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.161372472 (592h39m57.032455517s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:58.851873092 (592h39m58.722956137s) [upload=519577]: Next returned nil segment Dec 30 08:08:58.851904202 (592h39m58.722987247s) [upload=519577]: Waiting for error group managing segments... Dec 30 08:08:59.143407059 (592h39m59.014490104s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.143602201 (592h39m59.014685246s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.144760817 (592h39m59.015843852s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.153552237 (592h39m59.024635272s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.173075132 (592h39m59.044158177s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.230104896 (592h39m59.101187942s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238196233 (592h39m59.109279268s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238504911 (592h39m59.109587946s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.325614672 (592h39m59.196697717s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.439682385 (592h39m59.31076542s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.445870789 (592h39m59.316953854s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.467867354 (592h39m59.338950399s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494552707 (592h39m59.365635742s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494573135 (592h39m59.36565617s) [seg_pos={0 0}, upload=519577]: Segment reached optimal threshold of 13 pieces. Dec 30 08:08:59.494810718 (592h39m59.365893753s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection Dec 30 08:08:59.494817400 (592h39m59.365900436s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.494989989 (592h39m59.366073024s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection Dec 30 08:08:59.495013414 (592h39m59.366096459s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495030105 (592h39m59.36611314s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 08:08:59.495036769 (592h39m59.366119804s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495107033 (592h39m59.366190068s) [seg_pos={0 0}, upload=519577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:08:59.495174291 (592h39m59.366257326s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:08:59.495179631 (592h39m59.366262666s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:08:59.793154332 (592h39m59.664237367s) [upload=519577]: Done uploading segments... Dec 30 08:08:59.793190511 (592h39m59.664273546s) [upload=519577]: Upload finished. err: <nil> Duration: 5.741413044s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X...........] duration: 3.412230298s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:10:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.412256609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X....X.XX..X....XX.XX..................X...X..X..X..XX.X.X..] duration: 6.24880616s trace: https://jaeger.dev.storj.io/trace/3091684ebdfebfaf FAIL (2025-12-30 08:10:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ) context deadline exceeded ======LOGS====== egments... Dec 30 08:08:56.906787962 (592h39m56.777871007s) [upload=519577]: Got next segment. Inline: false Dec 30 08:08:56.906835392 (592h39m56.777918427s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:08:56.906849899 (592h39m56.777932945s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:08:57.151984305 (592h39m57.02306735s) [seg_pos={0 0}, upload=519577]: Begin upload segment... Dec 30 08:08:57.152045421 (592h39m57.023128466s) [seg_pos={0 0}, upload=519577]: Done begin upload segment. Dec 30 08:08:57.152088784 (592h39m57.023171829s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152103782 (592h39m57.023186817s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152948079 (592h39m57.024031114s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153318124 (592h39m57.024401159s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153610311 (592h39m57.024693346s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154041302 (592h39m57.025124337s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154116154 (592h39m57.025199199s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154499845 (592h39m57.02558289s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154851113 (592h39m57.025934158s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155328523 (592h39m57.026411558s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155630638 (592h39m57.026713673s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156234418 (592h39m57.027317463s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156721465 (592h39m57.02780451s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.158361338 (592h39m57.029444383s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.159690889 (592h39m57.030773924s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.161372472 (592h39m57.032455517s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:58.851873092 (592h39m58.722956137s) [upload=519577]: Next returned nil segment Dec 30 08:08:58.851904202 (592h39m58.722987247s) [upload=519577]: Waiting for error group managing segments... Dec 30 08:08:59.143407059 (592h39m59.014490104s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.143602201 (592h39m59.014685246s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.144760817 (592h39m59.015843852s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.153552237 (592h39m59.024635272s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.173075132 (592h39m59.044158177s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.230104896 (592h39m59.101187942s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238196233 (592h39m59.109279268s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238504911 (592h39m59.109587946s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.325614672 (592h39m59.196697717s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.439682385 (592h39m59.31076542s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.445870789 (592h39m59.316953854s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.467867354 (592h39m59.338950399s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494552707 (592h39m59.365635742s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494573135 (592h39m59.36565617s) [seg_pos={0 0}, upload=519577]: Segment reached optimal threshold of 13 pieces. Dec 30 08:08:59.494810718 (592h39m59.365893753s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection Dec 30 08:08:59.494817400 (592h39m59.365900436s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.494989989 (592h39m59.366073024s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection Dec 30 08:08:59.495013414 (592h39m59.366096459s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495030105 (592h39m59.36611314s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 08:08:59.495036769 (592h39m59.366119804s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495107033 (592h39m59.366190068s) [seg_pos={0 0}, upload=519577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:08:59.495174291 (592h39m59.366257326s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:08:59.495179631 (592h39m59.366262666s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:08:59.793154332 (592h39m59.664237367s) [upload=519577]: Done uploading segments... Dec 30 08:08:59.793190511 (592h39m59.664273546s) [upload=519577]: Upload finished. err: <nil> Duration: 6.248835626s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X..........] duration: 3.547848286s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:10:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547884135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X....X.XX..X....XX.XX..................X...X..X..X..XX.X.X.] duration: 5.832260519s trace: https://jaeger.dev.storj.io/trace/3091684ebdfebfaf FAIL (2025-12-30 08:10:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ) context deadline exceeded ======LOGS====== egments... Dec 30 08:08:56.906787962 (592h39m56.777871007s) [upload=519577]: Got next segment. Inline: false Dec 30 08:08:56.906835392 (592h39m56.777918427s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:08:56.906849899 (592h39m56.777932945s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:08:57.151984305 (592h39m57.02306735s) [seg_pos={0 0}, upload=519577]: Begin upload segment... Dec 30 08:08:57.152045421 (592h39m57.023128466s) [seg_pos={0 0}, upload=519577]: Done begin upload segment. Dec 30 08:08:57.152088784 (592h39m57.023171829s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152103782 (592h39m57.023186817s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152948079 (592h39m57.024031114s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153318124 (592h39m57.024401159s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153610311 (592h39m57.024693346s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154041302 (592h39m57.025124337s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154116154 (592h39m57.025199199s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154499845 (592h39m57.02558289s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154851113 (592h39m57.025934158s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155328523 (592h39m57.026411558s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155630638 (592h39m57.026713673s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156234418 (592h39m57.027317463s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156721465 (592h39m57.02780451s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.158361338 (592h39m57.029444383s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.159690889 (592h39m57.030773924s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.161372472 (592h39m57.032455517s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:58.851873092 (592h39m58.722956137s) [upload=519577]: Next returned nil segment Dec 30 08:08:58.851904202 (592h39m58.722987247s) [upload=519577]: Waiting for error group managing segments... Dec 30 08:08:59.143407059 (592h39m59.014490104s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.143602201 (592h39m59.014685246s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.144760817 (592h39m59.015843852s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.153552237 (592h39m59.024635272s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.173075132 (592h39m59.044158177s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.230104896 (592h39m59.101187942s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238196233 (592h39m59.109279268s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238504911 (592h39m59.109587946s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.325614672 (592h39m59.196697717s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.439682385 (592h39m59.31076542s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.445870789 (592h39m59.316953854s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.467867354 (592h39m59.338950399s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494552707 (592h39m59.365635742s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494573135 (592h39m59.36565617s) [seg_pos={0 0}, upload=519577]: Segment reached optimal threshold of 13 pieces. Dec 30 08:08:59.494810718 (592h39m59.365893753s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection Dec 30 08:08:59.494817400 (592h39m59.365900436s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.494989989 (592h39m59.366073024s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection Dec 30 08:08:59.495013414 (592h39m59.366096459s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495030105 (592h39m59.36611314s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 08:08:59.495036769 (592h39m59.366119804s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495107033 (592h39m59.366190068s) [seg_pos={0 0}, upload=519577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:08:59.495174291 (592h39m59.366257326s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:08:59.495179631 (592h39m59.366262666s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:08:59.793154332 (592h39m59.664237367s) [upload=519577]: Done uploading segments... Dec 30 08:08:59.793190511 (592h39m59.664273546s) [upload=519577]: Upload finished. err: <nil> Duration: 5.832307888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X................X....] duration: 15.126082533s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:09:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 15.126110316s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)