Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.......XXX..X.XX...X.....X.X...XX.XX...X....XX...XX.........] duration: 5.001924688s trace: https://jaeger.dev.storj.io/trace/724437a3d04dceb1 FAIL (2025-12-26 15:57:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001948945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X..................X..........X.XX.X....X....X..X...X......] duration: 58.411465497s trace: https://jaeger.dev.storj.io/trace/59cbf1a960844a99 FAIL (2025-12-26 15:57:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .986496058s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115512742 (504h22m13.986595777s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115811882 (504h22m13.986894917s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116226672 (504h22m13.987309717s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116713739 (504h22m13.987796794s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117256382 (504h22m13.988339427s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117915397 (504h22m13.988998432s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118334034 (504h22m13.989417079s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118780675 (504h22m13.98986371s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.119538688 (504h22m13.990621723s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.121375436 (504h22m13.992458471s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.122356614 (504h22m13.993439659s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123005210 (504h22m13.994088245s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123547612 (504h22m13.994630647s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124148125 (504h22m13.99523116s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124676171 (504h22m13.995759206s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:15.571926479 (504h22m15.443009514s) [upload=442593]: Next returned nil segment Dec 26 15:51:15.571941377 (504h22m15.443024412s) [upload=442593]: Waiting for error group managing segments... Dec 26 15:51:16.013260185 (504h22m15.884343231s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.236088980 (504h22m16.107172025s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.268119251 (504h22m16.139202296s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.294983495 (504h22m16.16606654s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.325102046 (504h22m16.196185081s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.342318109 (504h22m16.213401154s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.452085164 (504h22m16.323168209s) [noise=true , address=77.90.31.13:4633 , piece_id=IXI7FNQUXRBF6FLPRGFMRBWC65ECZKVFJ5BRODVMFML3RX24X4FA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.494318955 (504h22m16.365402s) [noise=true , address=77.90.31.17:4796 , piece_id=TPWPDCOKN3FJ3GEJBLVA7JS4MQXG7SVKKTMCGY3H2NVRSF2DNPFA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.724428072 (504h22m16.595511107s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.979014299 (504h22m16.850097334s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.984346784 (504h22m16.855429819s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.999686903 (504h22m16.870769958s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.251751369 (504h22m17.122834404s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385129784 (504h22m17.256212819s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385152197 (504h22m17.256235242s) [seg_pos={0 0}, upload=442593]: Segment reached optimal threshold of 14 pieces. Dec 26 15:51:17.385569151 (504h22m17.256652216s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385577687 (504h22m17.256660722s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385723565 (504h22m17.25680661s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385731430 (504h22m17.256814485s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385788949 (504h22m17.256871994s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection Dec 26 15:51:17.385944255 (504h22m17.25702731s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386455389 (504h22m17.257538434s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm): protocol: context canceled; context canceled Dec 26 15:51:17.386463554 (504h22m17.257546599s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386479724 (504h22m17.257562769s) [seg_pos={0 0}, upload=442593]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:51:17.386549227 (504h22m17.257632262s) [upload=442593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:51:17.386553525 (504h22m17.25763657s) [upload=442593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:51:17.689235945 (504h22m17.56031898s) [upload=442593]: Done uploading segments... Dec 26 15:51:17.689319835 (504h22m17.56040288s) [upload=442593]: Upload finished. err: <nil> Duration: 58.411484272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [........XXX..X.XX...X.....X.X...XX.XX...X....XX...XX........] duration: 3.498191096s trace: https://jaeger.dev.storj.io/trace/724437a3d04dceb1 FAIL (2025-12-26 15:57:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49821866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...XXX....XX...X......X..X..X.X......XX...................X.] duration: 6.135791458s trace: https://jaeger.dev.storj.io/trace/1001bf48696cdf04 FAIL (2025-12-26 15:57:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: QQNXYQVGBAA6W3OCUH6TLEATCQ7LZP7YZCW67E4ZKAMVWDU3I3YQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 4GZIUFWG4PLKAJIIOJ2POWHFB6GL77YUTKO276HJ4EYC6ZA2DCMA) context deadline exceeded ======LOGS====== Dec 26 15:56:21.639434421 (504h27m21.510517456s) [upload=442663]: Starting upload Dec 26 15:56:21.639520355 (504h27m21.5106034s) [upload=442663]: Done starting upload Dec 26 15:56:21.639607991 (504h27m21.510691026s) [upload=442663]: Uploading segments... Dec 26 15:56:21.639831127 (504h27m21.510914172s) [upload=442663]: Got next segment. Inline: false Dec 26 15:56:21.639893215 (504h27m21.51097625s) [upload=442663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:56:21.639903164 (504h27m21.510986239s) [upload=442663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:56:21.889970426 (504h27m21.761053461s) [seg_pos={0 0}, upload=442663]: Begin upload segment... Dec 26 15:56:21.890024338 (504h27m21.761107413s) [seg_pos={0 0}, upload=442663]: Done begin upload segment. Dec 26 15:56:21.890071638 (504h27m21.761154674s) [noise=true , address=145.79.227.41:4087 , piece_id=5LK2HCGFPHNV463SXQ6ECDS426XZFWSNCFUEWJRBIMTRFIB7EEEQ, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.890144327 (504h27m21.761227372s) [noise=true , address=145.79.227.14:4840 , piece_id=5S6ZA2CA7WFITKXBXTFCIHZOU4TTRMRR2MXCQUCBVQWSRRNV6SIA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.890831856 (504h27m21.761914891s) [noise=true , address=145.79.227.17:4970 , piece_id=RPXWXYLQJGAQ7W6TXPLRGDGN7HUET5LEU5TJK423FNEZWVWVFFAA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.890892441 (504h27m21.761975486s) [noise=true , address=77.90.31.11:4553 , piece_id=BQTWT3EZ7HNDMT5PT7UDMGWM3WUJNPZVNPYX53T6PH6WWXUQKARQ, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.891189647 (504h27m21.762272692s) [noise=true , address=145.79.227.4:4185 , piece_id=RLS2B7YNZKNWTQR2OXZ7LPZUZWEBJJTJ4TZX3JJ2S42ELFAD62MA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.891462036 (504h27m21.762545071s) [noise=true , address=145.79.227.45:4231 , piece_id=DV3BBXE3IJQBXHG2R64HEIMDFEIOYA56DXOAFCJXAANBGAA4CW3Q, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.891531068 (504h27m21.762614103s) [noise=true , address=145.79.227.48:4302 , piece_id=RUJXNKBKBTBLR6PGVR5SANVEH75K73X4FKQCIOC64WSP7XECR4FA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.891838483 (504h27m21.762921528s) [noise=true , address=145.79.227.19:4078 , piece_id=NWVTT6GRFBYHP7FDMCKRZC7C7IYVEIEOAFFJC3EAIPMCGKVWEDLQ, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.892015129 (504h27m21.763098174s) [noise=true , address=145.79.227.16:4910 , piece_id=LV7NMXEM6YIT54CIV7JYMTM3FVJQUNQFDKARLFXYTQLGL2WOREJA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.892144154 (504h27m21.763227189s) [noise=true , address=45.91.116.7:4009 , piece_id=YB6IULWYK457ZXIQ2BXHZ2W3CVL2ONJ75HJYDZHINB7JUNYND4UA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.892460116 (504h27m21.763543161s) [noise=true , address=145.79.227.22:4511 , piece_id=MNITPA4L4AQYUMUHJW2YUEKQ33ZSZDA4X2PZ52JNXCXGXENEF6UQ, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.892558063 (504h27m21.763641098s) [noise=true , address=145.79.227.13:4803 , piece_id=EYDY33B7WAVHQGE6JGXCT5WUHRWMSZ2QIC2JT64G5EWCHGPJIACA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.892767712 (504h27m21.763850747s) [noise=true , address=145.79.227.29:4943 , piece_id=S4Z66EFM37ONCNQBA4PVBNH44WKTW6GKKANCC4AW34U3HEFHSNSA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.893053937 (504h27m21.764137012s) [noise=true , address=77.90.31.18:4808 , piece_id=QQNXYQVGBAA6W3OCUH6TLEATCQ7LZP7YZCW67E4ZKAMVWDU3I3YQ, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.893100316 (504h27m21.764183361s) [noise=true , address=45.91.116.5:4299 , piece_id=HFKPPDLJ55PWIHD6EPZP2ZJI4KJXVWJMST65K7VWVETJFMW5UGBA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.893624213 (504h27m21.764707258s) [noise=true , address=45.91.116.10:4083 , piece_id=4GZIUFWG4PLKAJIIOJ2POWHFB6GL77YUTKO276HJ4EYC6ZA2DCMA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:23.048348519 (504h27m22.919431554s) [noise=true , address=77.90.31.11:4553 , piece_id=BQTWT3EZ7HNDMT5PT7UDMGWM3WUJNPZVNPYX53T6PH6WWXUQKARQ, seg_pos={0 0}, upload=442663]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 15:56:23.530767931 (504h27m23.401850976s) [upload=442663]: Next returned nil segment Dec 26 15:56:23.530781957 (504h27m23.401864992s) [upload=442663]: Waiting for error group managing segments... Dec 26 15:56:23.886721608 (504h27m23.757804643s) [noise=true , address=145.79.227.19:4078 , piece_id=NWVTT6GRFBYHP7FDMCKRZC7C7IYVEIEOAFFJC3EAIPMCGKVWEDLQ, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.900033356 (504h27m23.771116401s) [noise=true , address=145.79.227.29:4943 , piece_id=S4Z66EFM37ONCNQBA4PVBNH44WKTW6GKKANCC4AW34U3HEFHSNSA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.903366713 (504h27m23.774449758s) [noise=true , address=145.79.227.4:4185 , piece_id=RLS2B7YNZKNWTQR2OXZ7LPZUZWEBJJTJ4TZX3JJ2S42ELFAD62MA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.903947079 (504h27m23.775030124s) [noise=true , address=145.79.227.48:4302 , piece_id=RUJXNKBKBTBLR6PGVR5SANVEH75K73X4FKQCIOC64WSP7XECR4FA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.904430249 (504h27m23.775513284s) [noise=true , address=145.79.227.45:4231 , piece_id=DV3BBXE3IJQBXHG2R64HEIMDFEIOYA56DXOAFCJXAANBGAA4CW3Q, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.905681210 (504h27m23.776764245s) [noise=true , address=145.79.227.22:4511 , piece_id=MNITPA4L4AQYUMUHJW2YUEKQ33ZSZDA4X2PZ52JNXCXGXENEF6UQ, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.906713737 (504h27m23.777796772s) [noise=true , address=145.79.227.17:4970 , piece_id=RPXWXYLQJGAQ7W6TXPLRGDGN7HUET5LEU5TJK423FNEZWVWVFFAA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.911631130 (504h27m23.782714165s) [noise=true , address=145.79.227.14:4840 , piece_id=5S6ZA2CA7WFITKXBXTFCIHZOU4TTRMRR2MXCQUCBVQWSRRNV6SIA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.916142250 (504h27m23.787225285s) [noise=true , address=145.79.227.13:4803 , piece_id=EYDY33B7WAVHQGE6JGXCT5WUHRWMSZ2QIC2JT64G5EWCHGPJIACA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.919418619 (504h27m23.790501654s) [noise=true , address=145.79.227.16:4910 , piece_id=LV7NMXEM6YIT54CIV7JYMTM3FVJQUNQFDKARLFXYTQLGL2WOREJA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.922945935 (504h27m23.79402898s) [noise=true , address=145.79.227.41:4087 , piece_id=5LK2HCGFPHNV463SXQ6ECDS426XZFWSNCFUEWJRBIMTRFIB7EEEQ, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:24.165306622 (504h27m24.036389658s) [noise=true , address=45.91.116.10:4083 , piece_id=4GZIUFWG4PLKAJIIOJ2POWHFB6GL77YUTKO276HJ4EYC6ZA2DCMA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:24.238144459 (504h27m24.109227494s) [noise=true , address=77.90.31.18:4808 , piece_id=QQNXYQVGBAA6W3OCUH6TLEATCQ7LZP7YZCW67E4ZKAMVWDU3I3YQ, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:24.238171840 (504h27m24.109254885s) [seg_pos={0 0}, upload=442663]: Segment reached optimal threshold of 13 pieces. Dec 26 15:56:24.238416456 (504h27m24.109499491s) [noise=true , address=45.91.116.7:4009 , piece_id=YB6IULWYK457ZXIQ2BXHZ2W3CVL2ONJ75HJYDZHINB7JUNYND4UA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 15:56:24.238425644 (504h27m24.109508679s) [noise=true , address=45.91.116.7:4009 , piece_id=YB6IULWYK457ZXIQ2BXHZ2W3CVL2ONJ75HJYDZHINB7JUNYND4UA, seg_pos={0 0}, upload=442663]: Long tail cancellation due to optimal threshold Dec 26 15:56:24.238471882 (504h27m24.109554927s) [noise=true , address=45.91.116.5:4299 , piece_id=HFKPPDLJ55PWIHD6EPZP2ZJI4KJXVWJMST65K7VWVETJFMW5UGBA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 15:56:24.238480077 (504h27m24.109563112s) [noise=true , address=45.91.116.5:4299 , piece_id=HFKPPDLJ55PWIHD6EPZP2ZJI4KJXVWJMST65K7VWVETJFMW5UGBA, seg_pos={0 0}, upload=442663]: Long tail cancellation due to optimal threshold Dec 26 15:56:24.238496259 (504h27m24.109579294s) [seg_pos={0 0}, upload=442663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:56:24.238607570 (504h27m24.109690606s) [upload=442663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:56:24.238612810 (504h27m24.109695845s) [upload=442663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:56:24.545680822 (504h27m24.416763857s) [upload=442663]: Done uploading segments... Dec 26 15:56:24.545701831 (504h27m24.416784876s) [upload=442663]: Upload finished. err: <nil> Duration: 6.135819793s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....XXX....XX...X......X..X..X.X......XX...................X] duration: 30.002415507s trace: https://jaeger.dev.storj.io/trace/1001bf48696cdf04 FAIL (2025-12-26 15:56:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: QQNXYQVGBAA6W3OCUH6TLEATCQ7LZP7YZCW67E4ZKAMVWDU3I3YQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 4GZIUFWG4PLKAJIIOJ2POWHFB6GL77YUTKO276HJ4EYC6ZA2DCMA) context deadline exceeded ======LOGS====== Dec 26 15:56:21.639434421 (504h27m21.510517456s) [upload=442663]: Starting upload Dec 26 15:56:21.639520355 (504h27m21.5106034s) [upload=442663]: Done starting upload Dec 26 15:56:21.639607991 (504h27m21.510691026s) [upload=442663]: Uploading segments... Dec 26 15:56:21.639831127 (504h27m21.510914172s) [upload=442663]: Got next segment. Inline: false Dec 26 15:56:21.639893215 (504h27m21.51097625s) [upload=442663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:56:21.639903164 (504h27m21.510986239s) [upload=442663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:56:21.889970426 (504h27m21.761053461s) [seg_pos={0 0}, upload=442663]: Begin upload segment... Dec 26 15:56:21.890024338 (504h27m21.761107413s) [seg_pos={0 0}, upload=442663]: Done begin upload segment. Dec 26 15:56:21.890071638 (504h27m21.761154674s) [noise=true , address=145.79.227.41:4087 , piece_id=5LK2HCGFPHNV463SXQ6ECDS426XZFWSNCFUEWJRBIMTRFIB7EEEQ, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.890144327 (504h27m21.761227372s) [noise=true , address=145.79.227.14:4840 , piece_id=5S6ZA2CA7WFITKXBXTFCIHZOU4TTRMRR2MXCQUCBVQWSRRNV6SIA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.890831856 (504h27m21.761914891s) [noise=true , address=145.79.227.17:4970 , piece_id=RPXWXYLQJGAQ7W6TXPLRGDGN7HUET5LEU5TJK423FNEZWVWVFFAA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.890892441 (504h27m21.761975486s) [noise=true , address=77.90.31.11:4553 , piece_id=BQTWT3EZ7HNDMT5PT7UDMGWM3WUJNPZVNPYX53T6PH6WWXUQKARQ, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.891189647 (504h27m21.762272692s) [noise=true , address=145.79.227.4:4185 , piece_id=RLS2B7YNZKNWTQR2OXZ7LPZUZWEBJJTJ4TZX3JJ2S42ELFAD62MA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.891462036 (504h27m21.762545071s) [noise=true , address=145.79.227.45:4231 , piece_id=DV3BBXE3IJQBXHG2R64HEIMDFEIOYA56DXOAFCJXAANBGAA4CW3Q, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.891531068 (504h27m21.762614103s) [noise=true , address=145.79.227.48:4302 , piece_id=RUJXNKBKBTBLR6PGVR5SANVEH75K73X4FKQCIOC64WSP7XECR4FA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.891838483 (504h27m21.762921528s) [noise=true , address=145.79.227.19:4078 , piece_id=NWVTT6GRFBYHP7FDMCKRZC7C7IYVEIEOAFFJC3EAIPMCGKVWEDLQ, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.892015129 (504h27m21.763098174s) [noise=true , address=145.79.227.16:4910 , piece_id=LV7NMXEM6YIT54CIV7JYMTM3FVJQUNQFDKARLFXYTQLGL2WOREJA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.892144154 (504h27m21.763227189s) [noise=true , address=45.91.116.7:4009 , piece_id=YB6IULWYK457ZXIQ2BXHZ2W3CVL2ONJ75HJYDZHINB7JUNYND4UA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.892460116 (504h27m21.763543161s) [noise=true , address=145.79.227.22:4511 , piece_id=MNITPA4L4AQYUMUHJW2YUEKQ33ZSZDA4X2PZ52JNXCXGXENEF6UQ, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.892558063 (504h27m21.763641098s) [noise=true , address=145.79.227.13:4803 , piece_id=EYDY33B7WAVHQGE6JGXCT5WUHRWMSZ2QIC2JT64G5EWCHGPJIACA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.892767712 (504h27m21.763850747s) [noise=true , address=145.79.227.29:4943 , piece_id=S4Z66EFM37ONCNQBA4PVBNH44WKTW6GKKANCC4AW34U3HEFHSNSA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.893053937 (504h27m21.764137012s) [noise=true , address=77.90.31.18:4808 , piece_id=QQNXYQVGBAA6W3OCUH6TLEATCQ7LZP7YZCW67E4ZKAMVWDU3I3YQ, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.893100316 (504h27m21.764183361s) [noise=true , address=45.91.116.5:4299 , piece_id=HFKPPDLJ55PWIHD6EPZP2ZJI4KJXVWJMST65K7VWVETJFMW5UGBA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:21.893624213 (504h27m21.764707258s) [noise=true , address=45.91.116.10:4083 , piece_id=4GZIUFWG4PLKAJIIOJ2POWHFB6GL77YUTKO276HJ4EYC6ZA2DCMA, seg_pos={0 0}, upload=442663]: Uploading piece... Dec 26 15:56:23.048348519 (504h27m22.919431554s) [noise=true , address=77.90.31.11:4553 , piece_id=BQTWT3EZ7HNDMT5PT7UDMGWM3WUJNPZVNPYX53T6PH6WWXUQKARQ, seg_pos={0 0}, upload=442663]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 15:56:23.530767931 (504h27m23.401850976s) [upload=442663]: Next returned nil segment Dec 26 15:56:23.530781957 (504h27m23.401864992s) [upload=442663]: Waiting for error group managing segments... Dec 26 15:56:23.886721608 (504h27m23.757804643s) [noise=true , address=145.79.227.19:4078 , piece_id=NWVTT6GRFBYHP7FDMCKRZC7C7IYVEIEOAFFJC3EAIPMCGKVWEDLQ, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.900033356 (504h27m23.771116401s) [noise=true , address=145.79.227.29:4943 , piece_id=S4Z66EFM37ONCNQBA4PVBNH44WKTW6GKKANCC4AW34U3HEFHSNSA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.903366713 (504h27m23.774449758s) [noise=true , address=145.79.227.4:4185 , piece_id=RLS2B7YNZKNWTQR2OXZ7LPZUZWEBJJTJ4TZX3JJ2S42ELFAD62MA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.903947079 (504h27m23.775030124s) [noise=true , address=145.79.227.48:4302 , piece_id=RUJXNKBKBTBLR6PGVR5SANVEH75K73X4FKQCIOC64WSP7XECR4FA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.904430249 (504h27m23.775513284s) [noise=true , address=145.79.227.45:4231 , piece_id=DV3BBXE3IJQBXHG2R64HEIMDFEIOYA56DXOAFCJXAANBGAA4CW3Q, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.905681210 (504h27m23.776764245s) [noise=true , address=145.79.227.22:4511 , piece_id=MNITPA4L4AQYUMUHJW2YUEKQ33ZSZDA4X2PZ52JNXCXGXENEF6UQ, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.906713737 (504h27m23.777796772s) [noise=true , address=145.79.227.17:4970 , piece_id=RPXWXYLQJGAQ7W6TXPLRGDGN7HUET5LEU5TJK423FNEZWVWVFFAA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.911631130 (504h27m23.782714165s) [noise=true , address=145.79.227.14:4840 , piece_id=5S6ZA2CA7WFITKXBXTFCIHZOU4TTRMRR2MXCQUCBVQWSRRNV6SIA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.916142250 (504h27m23.787225285s) [noise=true , address=145.79.227.13:4803 , piece_id=EYDY33B7WAVHQGE6JGXCT5WUHRWMSZ2QIC2JT64G5EWCHGPJIACA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.919418619 (504h27m23.790501654s) [noise=true , address=145.79.227.16:4910 , piece_id=LV7NMXEM6YIT54CIV7JYMTM3FVJQUNQFDKARLFXYTQLGL2WOREJA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:23.922945935 (504h27m23.79402898s) [noise=true , address=145.79.227.41:4087 , piece_id=5LK2HCGFPHNV463SXQ6ECDS426XZFWSNCFUEWJRBIMTRFIB7EEEQ, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:24.165306622 (504h27m24.036389658s) [noise=true , address=45.91.116.10:4083 , piece_id=4GZIUFWG4PLKAJIIOJ2POWHFB6GL77YUTKO276HJ4EYC6ZA2DCMA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:24.238144459 (504h27m24.109227494s) [noise=true , address=77.90.31.18:4808 , piece_id=QQNXYQVGBAA6W3OCUH6TLEATCQ7LZP7YZCW67E4ZKAMVWDU3I3YQ, seg_pos={0 0}, upload=442663]: Done uploading piece. err: <nil> Dec 26 15:56:24.238171840 (504h27m24.109254885s) [seg_pos={0 0}, upload=442663]: Segment reached optimal threshold of 13 pieces. Dec 26 15:56:24.238416456 (504h27m24.109499491s) [noise=true , address=45.91.116.7:4009 , piece_id=YB6IULWYK457ZXIQ2BXHZ2W3CVL2ONJ75HJYDZHINB7JUNYND4UA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 15:56:24.238425644 (504h27m24.109508679s) [noise=true , address=45.91.116.7:4009 , piece_id=YB6IULWYK457ZXIQ2BXHZ2W3CVL2ONJ75HJYDZHINB7JUNYND4UA, seg_pos={0 0}, upload=442663]: Long tail cancellation due to optimal threshold Dec 26 15:56:24.238471882 (504h27m24.109554927s) [noise=true , address=45.91.116.5:4299 , piece_id=HFKPPDLJ55PWIHD6EPZP2ZJI4KJXVWJMST65K7VWVETJFMW5UGBA, seg_pos={0 0}, upload=442663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 15:56:24.238480077 (504h27m24.109563112s) [noise=true , address=45.91.116.5:4299 , piece_id=HFKPPDLJ55PWIHD6EPZP2ZJI4KJXVWJMST65K7VWVETJFMW5UGBA, seg_pos={0 0}, upload=442663]: Long tail cancellation due to optimal threshold Dec 26 15:56:24.238496259 (504h27m24.109579294s) [seg_pos={0 0}, upload=442663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:56:24.238607570 (504h27m24.109690606s) [upload=442663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:56:24.238612810 (504h27m24.109695845s) [upload=442663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:56:24.545680822 (504h27m24.416763857s) [upload=442663]: Done uploading segments... Dec 26 15:56:24.545701831 (504h27m24.416784876s) [upload=442663]: Upload finished. err: <nil> Duration: 30.002440013s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X..................X..........X.XX.X....X....X..X...X.....] duration: 43.909089121s trace: https://jaeger.dev.storj.io/trace/59cbf1a960844a99 FAIL (2025-12-26 15:56:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .986496058s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115512742 (504h22m13.986595777s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115811882 (504h22m13.986894917s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116226672 (504h22m13.987309717s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116713739 (504h22m13.987796794s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117256382 (504h22m13.988339427s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117915397 (504h22m13.988998432s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118334034 (504h22m13.989417079s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118780675 (504h22m13.98986371s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.119538688 (504h22m13.990621723s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.121375436 (504h22m13.992458471s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.122356614 (504h22m13.993439659s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123005210 (504h22m13.994088245s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123547612 (504h22m13.994630647s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124148125 (504h22m13.99523116s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124676171 (504h22m13.995759206s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:15.571926479 (504h22m15.443009514s) [upload=442593]: Next returned nil segment Dec 26 15:51:15.571941377 (504h22m15.443024412s) [upload=442593]: Waiting for error group managing segments... Dec 26 15:51:16.013260185 (504h22m15.884343231s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.236088980 (504h22m16.107172025s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.268119251 (504h22m16.139202296s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.294983495 (504h22m16.16606654s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.325102046 (504h22m16.196185081s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.342318109 (504h22m16.213401154s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.452085164 (504h22m16.323168209s) [noise=true , address=77.90.31.13:4633 , piece_id=IXI7FNQUXRBF6FLPRGFMRBWC65ECZKVFJ5BRODVMFML3RX24X4FA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.494318955 (504h22m16.365402s) [noise=true , address=77.90.31.17:4796 , piece_id=TPWPDCOKN3FJ3GEJBLVA7JS4MQXG7SVKKTMCGY3H2NVRSF2DNPFA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.724428072 (504h22m16.595511107s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.979014299 (504h22m16.850097334s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.984346784 (504h22m16.855429819s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.999686903 (504h22m16.870769958s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.251751369 (504h22m17.122834404s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385129784 (504h22m17.256212819s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385152197 (504h22m17.256235242s) [seg_pos={0 0}, upload=442593]: Segment reached optimal threshold of 14 pieces. Dec 26 15:51:17.385569151 (504h22m17.256652216s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385577687 (504h22m17.256660722s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385723565 (504h22m17.25680661s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385731430 (504h22m17.256814485s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385788949 (504h22m17.256871994s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection Dec 26 15:51:17.385944255 (504h22m17.25702731s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386455389 (504h22m17.257538434s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm): protocol: context canceled; context canceled Dec 26 15:51:17.386463554 (504h22m17.257546599s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386479724 (504h22m17.257562769s) [seg_pos={0 0}, upload=442593]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:51:17.386549227 (504h22m17.257632262s) [upload=442593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:51:17.386553525 (504h22m17.25763657s) [upload=442593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:51:17.689235945 (504h22m17.56031898s) [upload=442593]: Done uploading segments... Dec 26 15:51:17.689319835 (504h22m17.56040288s) [upload=442593]: Upload finished. err: <nil> Duration: 43.909101704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [X........XXX..X.XX...X.....X.X...XX.XX...X....XX...XX.......] duration: 5.001391763s trace: https://jaeger.dev.storj.io/trace/724437a3d04dceb1 FAIL (2025-12-26 15:56:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001419666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....XXX....XX...X......X..X..X.X......XX...................] duration: 6.136776385s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 6.13680054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......XXX....XX...X......X..X..X.X......XX..................] duration: 5.978463167s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:55:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.978491982s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX.X..................X..........X.XX.X....X....X..X...X....] duration: 37.372064806s trace: https://jaeger.dev.storj.io/trace/59cbf1a960844a99 FAIL (2025-12-26 15:55:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .986496058s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115512742 (504h22m13.986595777s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115811882 (504h22m13.986894917s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116226672 (504h22m13.987309717s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116713739 (504h22m13.987796794s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117256382 (504h22m13.988339427s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117915397 (504h22m13.988998432s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118334034 (504h22m13.989417079s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118780675 (504h22m13.98986371s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.119538688 (504h22m13.990621723s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.121375436 (504h22m13.992458471s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.122356614 (504h22m13.993439659s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123005210 (504h22m13.994088245s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123547612 (504h22m13.994630647s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124148125 (504h22m13.99523116s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124676171 (504h22m13.995759206s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:15.571926479 (504h22m15.443009514s) [upload=442593]: Next returned nil segment Dec 26 15:51:15.571941377 (504h22m15.443024412s) [upload=442593]: Waiting for error group managing segments... Dec 26 15:51:16.013260185 (504h22m15.884343231s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.236088980 (504h22m16.107172025s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.268119251 (504h22m16.139202296s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.294983495 (504h22m16.16606654s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.325102046 (504h22m16.196185081s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.342318109 (504h22m16.213401154s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.452085164 (504h22m16.323168209s) [noise=true , address=77.90.31.13:4633 , piece_id=IXI7FNQUXRBF6FLPRGFMRBWC65ECZKVFJ5BRODVMFML3RX24X4FA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.494318955 (504h22m16.365402s) [noise=true , address=77.90.31.17:4796 , piece_id=TPWPDCOKN3FJ3GEJBLVA7JS4MQXG7SVKKTMCGY3H2NVRSF2DNPFA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.724428072 (504h22m16.595511107s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.979014299 (504h22m16.850097334s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.984346784 (504h22m16.855429819s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.999686903 (504h22m16.870769958s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.251751369 (504h22m17.122834404s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385129784 (504h22m17.256212819s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385152197 (504h22m17.256235242s) [seg_pos={0 0}, upload=442593]: Segment reached optimal threshold of 14 pieces. Dec 26 15:51:17.385569151 (504h22m17.256652216s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385577687 (504h22m17.256660722s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385723565 (504h22m17.25680661s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385731430 (504h22m17.256814485s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385788949 (504h22m17.256871994s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection Dec 26 15:51:17.385944255 (504h22m17.25702731s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386455389 (504h22m17.257538434s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm): protocol: context canceled; context canceled Dec 26 15:51:17.386463554 (504h22m17.257546599s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386479724 (504h22m17.257562769s) [seg_pos={0 0}, upload=442593]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:51:17.386549227 (504h22m17.257632262s) [upload=442593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:51:17.386553525 (504h22m17.25763657s) [upload=442593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:51:17.689235945 (504h22m17.56031898s) [upload=442593]: Done uploading segments... Dec 26 15:51:17.689319835 (504h22m17.56040288s) [upload=442593]: Upload finished. err: <nil> Duration: 37.37208305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......XXX....XX...X......X..X..X.X......XX.................] duration: 6.050206479s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:55:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 6.050236777s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.X........XXX..X.XX...X.....X.X...XX.XX...X....XX...XX......] duration: 3.388235732s trace: https://jaeger.dev.storj.io/trace/724437a3d04dceb1 FAIL (2025-12-26 15:55:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.388269296s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXX.X..................X..........X.XX.X....X....X..X...X...] duration: 59.945491691s trace: https://jaeger.dev.storj.io/trace/59cbf1a960844a99 FAIL (2025-12-26 15:54:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .986496058s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115512742 (504h22m13.986595777s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115811882 (504h22m13.986894917s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116226672 (504h22m13.987309717s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116713739 (504h22m13.987796794s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117256382 (504h22m13.988339427s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117915397 (504h22m13.988998432s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118334034 (504h22m13.989417079s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118780675 (504h22m13.98986371s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.119538688 (504h22m13.990621723s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.121375436 (504h22m13.992458471s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.122356614 (504h22m13.993439659s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123005210 (504h22m13.994088245s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123547612 (504h22m13.994630647s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124148125 (504h22m13.99523116s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124676171 (504h22m13.995759206s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:15.571926479 (504h22m15.443009514s) [upload=442593]: Next returned nil segment Dec 26 15:51:15.571941377 (504h22m15.443024412s) [upload=442593]: Waiting for error group managing segments... Dec 26 15:51:16.013260185 (504h22m15.884343231s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.236088980 (504h22m16.107172025s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.268119251 (504h22m16.139202296s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.294983495 (504h22m16.16606654s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.325102046 (504h22m16.196185081s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.342318109 (504h22m16.213401154s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.452085164 (504h22m16.323168209s) [noise=true , address=77.90.31.13:4633 , piece_id=IXI7FNQUXRBF6FLPRGFMRBWC65ECZKVFJ5BRODVMFML3RX24X4FA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.494318955 (504h22m16.365402s) [noise=true , address=77.90.31.17:4796 , piece_id=TPWPDCOKN3FJ3GEJBLVA7JS4MQXG7SVKKTMCGY3H2NVRSF2DNPFA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.724428072 (504h22m16.595511107s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.979014299 (504h22m16.850097334s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.984346784 (504h22m16.855429819s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.999686903 (504h22m16.870769958s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.251751369 (504h22m17.122834404s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385129784 (504h22m17.256212819s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385152197 (504h22m17.256235242s) [seg_pos={0 0}, upload=442593]: Segment reached optimal threshold of 14 pieces. Dec 26 15:51:17.385569151 (504h22m17.256652216s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385577687 (504h22m17.256660722s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385723565 (504h22m17.25680661s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385731430 (504h22m17.256814485s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385788949 (504h22m17.256871994s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection Dec 26 15:51:17.385944255 (504h22m17.25702731s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386455389 (504h22m17.257538434s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm): protocol: context canceled; context canceled Dec 26 15:51:17.386463554 (504h22m17.257546599s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386479724 (504h22m17.257562769s) [seg_pos={0 0}, upload=442593]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:51:17.386549227 (504h22m17.257632262s) [upload=442593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:51:17.386553525 (504h22m17.25763657s) [upload=442593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:51:17.689235945 (504h22m17.56031898s) [upload=442593]: Done uploading segments... Dec 26 15:51:17.689319835 (504h22m17.56040288s) [upload=442593]: Upload finished. err: <nil> Duration: 59.945511808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........XXX....XX...X......X..X..X.X......XX................] duration: 6.33649307s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:54:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 6.336520461s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X.X........XXX..X.XX...X.....X.X...XX.XX...X....XX...XX.....] duration: 4.789129388s trace: https://jaeger.dev.storj.io/trace/724437a3d04dceb1 FAIL (2025-12-26 15:54:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.789153815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........XXX....XX...X......X..X..X.X......XX...............] duration: 5.797755559s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:54:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.797783231s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X.X........XXX..X.XX...X.....X.X...XX.XX...X....XX...XX....] duration: 5.001192053s trace: https://jaeger.dev.storj.io/trace/724437a3d04dceb1 FAIL (2025-12-26 15:54:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001219025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXXX.X..................X..........X.XX.X....X....X..X...X..] duration: 56.8203796s trace: https://jaeger.dev.storj.io/trace/59cbf1a960844a99 FAIL (2025-12-26 15:53:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .986496058s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115512742 (504h22m13.986595777s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115811882 (504h22m13.986894917s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116226672 (504h22m13.987309717s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116713739 (504h22m13.987796794s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117256382 (504h22m13.988339427s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117915397 (504h22m13.988998432s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118334034 (504h22m13.989417079s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118780675 (504h22m13.98986371s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.119538688 (504h22m13.990621723s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.121375436 (504h22m13.992458471s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.122356614 (504h22m13.993439659s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123005210 (504h22m13.994088245s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123547612 (504h22m13.994630647s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124148125 (504h22m13.99523116s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124676171 (504h22m13.995759206s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:15.571926479 (504h22m15.443009514s) [upload=442593]: Next returned nil segment Dec 26 15:51:15.571941377 (504h22m15.443024412s) [upload=442593]: Waiting for error group managing segments... Dec 26 15:51:16.013260185 (504h22m15.884343231s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.236088980 (504h22m16.107172025s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.268119251 (504h22m16.139202296s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.294983495 (504h22m16.16606654s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.325102046 (504h22m16.196185081s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.342318109 (504h22m16.213401154s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.452085164 (504h22m16.323168209s) [noise=true , address=77.90.31.13:4633 , piece_id=IXI7FNQUXRBF6FLPRGFMRBWC65ECZKVFJ5BRODVMFML3RX24X4FA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.494318955 (504h22m16.365402s) [noise=true , address=77.90.31.17:4796 , piece_id=TPWPDCOKN3FJ3GEJBLVA7JS4MQXG7SVKKTMCGY3H2NVRSF2DNPFA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.724428072 (504h22m16.595511107s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.979014299 (504h22m16.850097334s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.984346784 (504h22m16.855429819s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.999686903 (504h22m16.870769958s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.251751369 (504h22m17.122834404s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385129784 (504h22m17.256212819s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385152197 (504h22m17.256235242s) [seg_pos={0 0}, upload=442593]: Segment reached optimal threshold of 14 pieces. Dec 26 15:51:17.385569151 (504h22m17.256652216s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385577687 (504h22m17.256660722s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385723565 (504h22m17.25680661s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385731430 (504h22m17.256814485s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385788949 (504h22m17.256871994s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection Dec 26 15:51:17.385944255 (504h22m17.25702731s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386455389 (504h22m17.257538434s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm): protocol: context canceled; context canceled Dec 26 15:51:17.386463554 (504h22m17.257546599s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386479724 (504h22m17.257562769s) [seg_pos={0 0}, upload=442593]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:51:17.386549227 (504h22m17.257632262s) [upload=442593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:51:17.386553525 (504h22m17.25763657s) [upload=442593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:51:17.689235945 (504h22m17.56031898s) [upload=442593]: Done uploading segments... Dec 26 15:51:17.689319835 (504h22m17.56040288s) [upload=442593]: Upload finished. err: <nil> Duration: 56.820394167s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X.X.X........XXX..X.XX...X.....X.X...XX.XX...X....XX...XX...] duration: 3.420444012s trace: https://jaeger.dev.storj.io/trace/724437a3d04dceb1 FAIL (2025-12-26 15:53:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420499067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........XXX....XX...X......X..X..X.X......XX..............] duration: 5.83745339s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:53:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.837488477s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X.X.X........XXX..X.XX...X.....X.X...XX.XX...X....XX...XX..] duration: 3.822185668s trace: https://jaeger.dev.storj.io/trace/724437a3d04dceb1 FAIL (2025-12-26 15:52:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.822203663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XXXX.X..................X..........X.XX.X....X....X..X...X.] duration: 45.332405396s trace: https://jaeger.dev.storj.io/trace/59cbf1a960844a99 FAIL (2025-12-26 15:52:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .986496058s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115512742 (504h22m13.986595777s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115811882 (504h22m13.986894917s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116226672 (504h22m13.987309717s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116713739 (504h22m13.987796794s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117256382 (504h22m13.988339427s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117915397 (504h22m13.988998432s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118334034 (504h22m13.989417079s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118780675 (504h22m13.98986371s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.119538688 (504h22m13.990621723s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.121375436 (504h22m13.992458471s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.122356614 (504h22m13.993439659s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123005210 (504h22m13.994088245s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123547612 (504h22m13.994630647s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124148125 (504h22m13.99523116s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124676171 (504h22m13.995759206s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:15.571926479 (504h22m15.443009514s) [upload=442593]: Next returned nil segment Dec 26 15:51:15.571941377 (504h22m15.443024412s) [upload=442593]: Waiting for error group managing segments... Dec 26 15:51:16.013260185 (504h22m15.884343231s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.236088980 (504h22m16.107172025s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.268119251 (504h22m16.139202296s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.294983495 (504h22m16.16606654s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.325102046 (504h22m16.196185081s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.342318109 (504h22m16.213401154s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.452085164 (504h22m16.323168209s) [noise=true , address=77.90.31.13:4633 , piece_id=IXI7FNQUXRBF6FLPRGFMRBWC65ECZKVFJ5BRODVMFML3RX24X4FA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.494318955 (504h22m16.365402s) [noise=true , address=77.90.31.17:4796 , piece_id=TPWPDCOKN3FJ3GEJBLVA7JS4MQXG7SVKKTMCGY3H2NVRSF2DNPFA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.724428072 (504h22m16.595511107s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.979014299 (504h22m16.850097334s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.984346784 (504h22m16.855429819s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.999686903 (504h22m16.870769958s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.251751369 (504h22m17.122834404s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385129784 (504h22m17.256212819s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385152197 (504h22m17.256235242s) [seg_pos={0 0}, upload=442593]: Segment reached optimal threshold of 14 pieces. Dec 26 15:51:17.385569151 (504h22m17.256652216s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385577687 (504h22m17.256660722s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385723565 (504h22m17.25680661s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385731430 (504h22m17.256814485s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385788949 (504h22m17.256871994s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection Dec 26 15:51:17.385944255 (504h22m17.25702731s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386455389 (504h22m17.257538434s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm): protocol: context canceled; context canceled Dec 26 15:51:17.386463554 (504h22m17.257546599s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386479724 (504h22m17.257562769s) [seg_pos={0 0}, upload=442593]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:51:17.386549227 (504h22m17.257632262s) [upload=442593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:51:17.386553525 (504h22m17.25763657s) [upload=442593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:51:17.689235945 (504h22m17.56031898s) [upload=442593]: Done uploading segments... Dec 26 15:51:17.689319835 (504h22m17.56040288s) [upload=442593]: Upload finished. err: <nil> Duration: 45.332421686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...........XXX....XX...X......X..X..X.X......XX.............] duration: 5.75374755s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:52:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.753777056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.XXXX.X..................X..........X.XX.X....X....X..X...X] duration: 1m0.003289317s trace: https://jaeger.dev.storj.io/trace/59cbf1a960844a99 FAIL (2025-12-26 15:52:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .986496058s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115512742 (504h22m13.986595777s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.115811882 (504h22m13.986894917s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116226672 (504h22m13.987309717s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.116713739 (504h22m13.987796794s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117256382 (504h22m13.988339427s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.117915397 (504h22m13.988998432s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118334034 (504h22m13.989417079s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.118780675 (504h22m13.98986371s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.119538688 (504h22m13.990621723s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.121375436 (504h22m13.992458471s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.122356614 (504h22m13.993439659s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123005210 (504h22m13.994088245s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.123547612 (504h22m13.994630647s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124148125 (504h22m13.99523116s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:14.124676171 (504h22m13.995759206s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Uploading piece... Dec 26 15:51:15.571926479 (504h22m15.443009514s) [upload=442593]: Next returned nil segment Dec 26 15:51:15.571941377 (504h22m15.443024412s) [upload=442593]: Waiting for error group managing segments... Dec 26 15:51:16.013260185 (504h22m15.884343231s) [noise=true , address=145.79.227.52:4449 , piece_id=7VYDAKH6NBLWMR6MSCHLLI46G3QFLJG7I4RR5Q4XJYEAGSIHXYRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.236088980 (504h22m16.107172025s) [noise=true , address=45.91.116.9:4276 , piece_id=PJCCYGKJZ362WGDYJJN2O5EAG5FDM24QCJ2STWZWEXSBJHZH6N4Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.268119251 (504h22m16.139202296s) [noise=true , address=64.49.13.26:4945 , piece_id=XAAHE5XHS4JPMRGLJDS4LTXR4JNHPWZLURU6ZU7SHUWZCJ72Y73Q, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.294983495 (504h22m16.16606654s) [noise=true , address=77.90.31.16:4765 , piece_id=75QW7OFDRP5EJJ6AEW3LO3O7OC762XHCW3KVX7AFQV5LVKLF4NRA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.325102046 (504h22m16.196185081s) [noise=true , address=45.91.116.4:4965 , piece_id=JTVIZI4KW27KWB35H5PHDYYCZ73DS6JY3DIIJ6KP45IRVAS4U7WQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.342318109 (504h22m16.213401154s) [noise=true , address=77.90.31.18:4830 , piece_id=JJOMYMQZBQSKDQLSDUCVEN5OVMWLJFE3DNK4BVWE47EGGFYIN4TQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.452085164 (504h22m16.323168209s) [noise=true , address=77.90.31.13:4633 , piece_id=IXI7FNQUXRBF6FLPRGFMRBWC65ECZKVFJ5BRODVMFML3RX24X4FA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.494318955 (504h22m16.365402s) [noise=true , address=77.90.31.17:4796 , piece_id=TPWPDCOKN3FJ3GEJBLVA7JS4MQXG7SVKKTMCGY3H2NVRSF2DNPFA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.724428072 (504h22m16.595511107s) [noise=true , address=193.149.164.143:4700 , piece_id=DC3XRC4VOLTPQUJKJJIZTI3VV5T3IQYQNMHHKYUPIIVTM6WVCJDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.979014299 (504h22m16.850097334s) [noise=true , address=145.79.227.44:4217 , piece_id=UBENAKC5YWMP7UMLMEM5VV7DXGKRYIVUVRNXACKQPWACV5GXRVVA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.984346784 (504h22m16.855429819s) [noise=true , address=145.79.227.30:4773 , piece_id=Y4USF7LH2OGZMRL63N6FZWVLJPWO2FEGKBWNDJ33U4A3DZZAKCSQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:16.999686903 (504h22m16.870769958s) [noise=true , address=145.79.227.34:4918 , piece_id=7TYZQOSY5MWZHKTT2RWKNVZHU7TGZ3CNZKFS43ZJ7KGMNHB675IA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.251751369 (504h22m17.122834404s) [noise=true , address=77.90.31.19:4854 , piece_id=ZPQDBFLQD7VF5IZOD42AYT2XDZS2BL7MCJMEGODPWU3DMND3B3XA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385129784 (504h22m17.256212819s) [noise=true , address=77.90.31.15:4721 , piece_id=PK4H5FLCIZFQ5IMDNW6TYZXA7LNAE47YHY6MIHAU35DWNYSGYZBA, seg_pos={0 0}, upload=442593]: Done uploading piece. err: <nil> Dec 26 15:51:17.385152197 (504h22m17.256235242s) [seg_pos={0 0}, upload=442593]: Segment reached optimal threshold of 14 pieces. Dec 26 15:51:17.385569151 (504h22m17.256652216s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385577687 (504h22m17.256660722s) [noise=true , address=77.90.31.10:4549 , piece_id=ZHZWHPTEZ6FLM2WJIZEFT237AK2KX7NA5ZOJQCXRW35GATNL3DDQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385723565 (504h22m17.25680661s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 26 15:51:17.385731430 (504h22m17.256814485s) [noise=true , address=145.79.227.40:4062 , piece_id=26QUGSOYN4FKJDRD243S3MXTBVEDE2KSGETJNX4ZZEKQIL53D77A, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.385788949 (504h22m17.256871994s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35202->64.49.13.169:4708: use of closed network connection Dec 26 15:51:17.385944255 (504h22m17.25702731s) [noise=true , address=64.49.13.169:4708 , piece_id=6ZG37BONUXVFTT2FC4YPPZNBD36WQV42BLN4NHFRZMYOJQSLSFYQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386455389 (504h22m17.257538434s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm): protocol: context canceled; context canceled Dec 26 15:51:17.386463554 (504h22m17.257546599s) [noise=true , address=45.91.116.5:4327 , piece_id=NTPVDP5LSIUWCTEVQGMG3SFSDVS7NNY336M4354M6FVPV7AXEXGQ, seg_pos={0 0}, upload=442593]: Long tail cancellation due to optimal threshold Dec 26 15:51:17.386479724 (504h22m17.257562769s) [seg_pos={0 0}, upload=442593]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:51:17.386549227 (504h22m17.257632262s) [upload=442593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:51:17.386553525 (504h22m17.25763657s) [upload=442593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:51:17.689235945 (504h22m17.56031898s) [upload=442593]: Done uploading segments... Dec 26 15:51:17.689319835 (504h22m17.56040288s) [upload=442593]: Upload finished. err: <nil> Duration: 1m0.003315738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [..X.X.X........XXX..X.XX...X.....X.X...XX.XX...X....XX...XX.] duration: 3.361972683s trace: https://jaeger.dev.storj.io/trace/724437a3d04dceb1 FAIL (2025-12-26 15:52:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.362000436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............XXX....XX...X......X..X..X.X......XX............] duration: 5.765833724s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:52:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.765879892s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X....XX...XX] duration: 5.001160493s trace: https://jaeger.dev.storj.io/trace/724437a3d04dceb1 FAIL (2025-12-26 15:51:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001183135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X............XXX....XX...X......X..X..X.X......XX...........] duration: 5.744120859s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:51:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.744164893s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X....XX...X] duration: 5.001513214s trace: https://jaeger.dev.storj.io/trace/3865259a052201a FAIL (2025-12-26 15:51:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001533302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X............XXX....XX...X......X..X..X.X......XX..........] duration: 5.750404533s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:50:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.750457565s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X............XXX....XX...X......X..X..X.X......XX.........] duration: 5.823067056s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:50:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.823087245s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX.XXXX.X..................X..........X.XX.X....X....X..X...] duration: 25.070277525s trace: https://jaeger.dev.storj.io/trace/5e07f788228f512 FAIL (2025-12-26 15:50:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (504h18m2.510656332s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.639766805 (504h18m2.51084984s) [noise=true , address=77.90.31.19:4854 , piece_id=UCSS6VRDOAD2C46W4FZ6JY2Y37H7N7H5R3COTXZVG3RMJLSKUUFA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.639948822 (504h18m2.511031857s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640103847 (504h18m2.511186892s) [noise=true , address=45.91.116.10:4092 , piece_id=4PC6UULS6ORGKBKLAU252LL3TEIPG33NBJQYPUHHZACUVTD6ADPA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640330458 (504h18m2.511413503s) [noise=true , address=193.149.164.143:4701 , piece_id=O6VGCDFTSGYTN6F7D72K6JOA667ZMFV3O6DRKCYXJEL6NJFNCFWQ, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640555597 (504h18m2.511638633s) [noise=true , address=145.79.227.30:4777 , piece_id=VCYPI4657CWNMMNYGXEUIVCOTJY3GSEI3F7GKQ546QIEAWLE2ZEA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640738696 (504h18m2.511821761s) [noise=true , address=77.90.31.18:4821 , piece_id=VBTID3ZSFHPFRDO7LDMBHB4UGZ5TMXITVRGPFKGVDMGIMXORC3CA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640904802 (504h18m2.511987837s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641146643 (504h18m2.512229678s) [noise=true , address=145.79.227.40:4083 , piece_id=QYVAADWLI7ZNMN6N5SAPA35CWS5Y6NZYKL7LLKWIFLMCYYKXH4UA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641438939 (504h18m2.512521984s) [noise=true , address=145.79.227.34:4936 , piece_id=P6YGSLH57DM2W47VSRJVND3WVVDHTS4ODEBMFIJNHTZGDUSESSBA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641601249 (504h18m2.512684284s) [noise=true , address=145.79.227.33:4885 , piece_id=UXXEZZTTTHZJFLR2ZCWFA4SR2IDF2TMPOATWRXNMQDML77FZLS3A, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641873547 (504h18m2.512956582s) [noise=true , address=45.91.116.13:4165 , piece_id=SMSJAQ27WZIXIBJHG2VB42XZXUEI7VGC5VRVOH7PXNDHUH5WETPA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642045344 (504h18m2.51312839s) [noise=true , address=77.90.31.15:4706 , piece_id=EQF53SQSFEXSYKKJYJKPK43E2WKZ5LD4MEWXH3N54V46AZUYCJYA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642234935 (504h18m2.51331798s) [noise=true , address=145.79.227.14:4850 , piece_id=HWBSZFDBWYAAGQTLXBW3ZNPMUAFC3U7KXARKCYQHPCC5E55AEZJA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642576606 (504h18m2.513659651s) [noise=true , address=77.90.31.12:4597 , piece_id=JHVQ75DVU5EPWOJXNKS24424XANPKG4OM3V2BEZFNKQEGVCERKZA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:03.955474533 (504h18m3.826557578s) [upload=442530]: Next returned nil segment Dec 26 15:47:03.955496284 (504h18m3.826579319s) [upload=442530]: Waiting for error group managing segments... Dec 26 15:47:04.292162345 (504h18m4.1632454s) [noise=true , address=193.149.164.143:4701 , piece_id=O6VGCDFTSGYTN6F7D72K6JOA667ZMFV3O6DRKCYXJEL6NJFNCFWQ, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.573847513 (504h18m4.444930558s) [noise=true , address=145.79.227.14:4850 , piece_id=HWBSZFDBWYAAGQTLXBW3ZNPMUAFC3U7KXARKCYQHPCC5E55AEZJA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.692327759 (504h18m4.563410804s) [noise=true , address=77.90.31.13:4656 , piece_id=3HFWM7ORA6VYY5XMGM5GHHQ5YAELNK6DLZUOQEQQ5WSL6KDGMOQA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.816015032 (504h18m4.687098067s) [noise=true , address=77.90.31.15:4706 , piece_id=EQF53SQSFEXSYKKJYJKPK43E2WKZ5LD4MEWXH3N54V46AZUYCJYA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.878724030 (504h18m4.749807075s) [noise=true , address=77.90.31.19:4854 , piece_id=UCSS6VRDOAD2C46W4FZ6JY2Y37H7N7H5R3COTXZVG3RMJLSKUUFA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.906686897 (504h18m4.777769932s) [noise=true , address=77.90.31.18:4821 , piece_id=VBTID3ZSFHPFRDO7LDMBHB4UGZ5TMXITVRGPFKGVDMGIMXORC3CA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.113831026 (504h18m4.984914061s) [noise=true , address=77.90.31.12:4597 , piece_id=JHVQ75DVU5EPWOJXNKS24424XANPKG4OM3V2BEZFNKQEGVCERKZA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.217449733 (504h18m5.088532778s) [noise=true , address=45.91.116.10:4092 , piece_id=4PC6UULS6ORGKBKLAU252LL3TEIPG33NBJQYPUHHZACUVTD6ADPA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.443177628 (504h18m5.314260663s) [noise=true , address=145.79.227.44:4205 , piece_id=XBCTB3HFCLOY3PWYANKNF3XTITDFL2M2GN436Y3WFEKCWBY7K2BQ, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.460351762 (504h18m5.331434807s) [noise=true , address=145.79.227.30:4777 , piece_id=VCYPI4657CWNMMNYGXEUIVCOTJY3GSEI3F7GKQ546QIEAWLE2ZEA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.495633474 (504h18m5.36671652s) [noise=true , address=145.79.227.34:4936 , piece_id=P6YGSLH57DM2W47VSRJVND3WVVDHTS4ODEBMFIJNHTZGDUSESSBA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.512258741 (504h18m5.383341786s) [noise=true , address=145.79.227.40:4083 , piece_id=QYVAADWLI7ZNMN6N5SAPA35CWS5Y6NZYKL7LLKWIFLMCYYKXH4UA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.622007582 (504h18m5.493090637s) [noise=true , address=145.79.227.33:4885 , piece_id=UXXEZZTTTHZJFLR2ZCWFA4SR2IDF2TMPOATWRXNMQDML77FZLS3A, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.828010718 (504h18m5.699093763s) [noise=true , address=45.91.116.13:4165 , piece_id=SMSJAQ27WZIXIBJHG2VB42XZXUEI7VGC5VRVOH7PXNDHUH5WETPA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.828033271 (504h18m5.699116316s) [seg_pos={0 0}, upload=442530]: Segment reached optimal threshold of 14 pieces. Dec 26 15:47:05.828240316 (504h18m5.699323361s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection Dec 26 15:47:05.828247699 (504h18m5.699330745s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828383899 (504h18m5.699466944s) [noise=true , address=77.90.31.11:4562 , piece_id=2RIE4646FNTIN6NFI6NJBI3VDDUFW2RQZ5LTZQVHBN6JPCISMWKA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 26 15:47:05.828390852 (504h18m5.699473897s) [noise=true , address=77.90.31.11:4562 , piece_id=2RIE4646FNTIN6NFI6NJBI3VDDUFW2RQZ5LTZQVHBN6JPCISMWKA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828580433 (504h18m5.699663499s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh): protocol: write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection; write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection Dec 26 15:47:05.828607355 (504h18m5.69969039s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828998109 (504h18m5.700081154s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XwcXkyKfaFNtGQpfU6npzHYAz2ySySmvdT9ya9HUa7u6dR2Zy): protocol: context canceled; context canceled Dec 26 15:47:05.829053253 (504h18m5.700136288s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.829080796 (504h18m5.700163841s) [seg_pos={0 0}, upload=442530]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:47:05.829185195 (504h18m5.70026824s) [upload=442530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:47:05.829190374 (504h18m5.700273419s) [upload=442530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:47:06.133692099 (504h18m6.004775144s) [upload=442530]: Done uploading segments... Dec 26 15:47:06.133710274 (504h18m6.004793319s) [upload=442530]: Upload finished. err: <nil> Duration: 25.070297732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX.XXXX.X..................X..........X.XX.X....X....X..X..] duration: 45.807036794s trace: https://jaeger.dev.storj.io/trace/5e07f788228f512 FAIL (2025-12-26 15:49:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (504h18m2.510656332s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.639766805 (504h18m2.51084984s) [noise=true , address=77.90.31.19:4854 , piece_id=UCSS6VRDOAD2C46W4FZ6JY2Y37H7N7H5R3COTXZVG3RMJLSKUUFA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.639948822 (504h18m2.511031857s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640103847 (504h18m2.511186892s) [noise=true , address=45.91.116.10:4092 , piece_id=4PC6UULS6ORGKBKLAU252LL3TEIPG33NBJQYPUHHZACUVTD6ADPA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640330458 (504h18m2.511413503s) [noise=true , address=193.149.164.143:4701 , piece_id=O6VGCDFTSGYTN6F7D72K6JOA667ZMFV3O6DRKCYXJEL6NJFNCFWQ, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640555597 (504h18m2.511638633s) [noise=true , address=145.79.227.30:4777 , piece_id=VCYPI4657CWNMMNYGXEUIVCOTJY3GSEI3F7GKQ546QIEAWLE2ZEA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640738696 (504h18m2.511821761s) [noise=true , address=77.90.31.18:4821 , piece_id=VBTID3ZSFHPFRDO7LDMBHB4UGZ5TMXITVRGPFKGVDMGIMXORC3CA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640904802 (504h18m2.511987837s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641146643 (504h18m2.512229678s) [noise=true , address=145.79.227.40:4083 , piece_id=QYVAADWLI7ZNMN6N5SAPA35CWS5Y6NZYKL7LLKWIFLMCYYKXH4UA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641438939 (504h18m2.512521984s) [noise=true , address=145.79.227.34:4936 , piece_id=P6YGSLH57DM2W47VSRJVND3WVVDHTS4ODEBMFIJNHTZGDUSESSBA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641601249 (504h18m2.512684284s) [noise=true , address=145.79.227.33:4885 , piece_id=UXXEZZTTTHZJFLR2ZCWFA4SR2IDF2TMPOATWRXNMQDML77FZLS3A, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641873547 (504h18m2.512956582s) [noise=true , address=45.91.116.13:4165 , piece_id=SMSJAQ27WZIXIBJHG2VB42XZXUEI7VGC5VRVOH7PXNDHUH5WETPA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642045344 (504h18m2.51312839s) [noise=true , address=77.90.31.15:4706 , piece_id=EQF53SQSFEXSYKKJYJKPK43E2WKZ5LD4MEWXH3N54V46AZUYCJYA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642234935 (504h18m2.51331798s) [noise=true , address=145.79.227.14:4850 , piece_id=HWBSZFDBWYAAGQTLXBW3ZNPMUAFC3U7KXARKCYQHPCC5E55AEZJA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642576606 (504h18m2.513659651s) [noise=true , address=77.90.31.12:4597 , piece_id=JHVQ75DVU5EPWOJXNKS24424XANPKG4OM3V2BEZFNKQEGVCERKZA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:03.955474533 (504h18m3.826557578s) [upload=442530]: Next returned nil segment Dec 26 15:47:03.955496284 (504h18m3.826579319s) [upload=442530]: Waiting for error group managing segments... Dec 26 15:47:04.292162345 (504h18m4.1632454s) [noise=true , address=193.149.164.143:4701 , piece_id=O6VGCDFTSGYTN6F7D72K6JOA667ZMFV3O6DRKCYXJEL6NJFNCFWQ, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.573847513 (504h18m4.444930558s) [noise=true , address=145.79.227.14:4850 , piece_id=HWBSZFDBWYAAGQTLXBW3ZNPMUAFC3U7KXARKCYQHPCC5E55AEZJA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.692327759 (504h18m4.563410804s) [noise=true , address=77.90.31.13:4656 , piece_id=3HFWM7ORA6VYY5XMGM5GHHQ5YAELNK6DLZUOQEQQ5WSL6KDGMOQA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.816015032 (504h18m4.687098067s) [noise=true , address=77.90.31.15:4706 , piece_id=EQF53SQSFEXSYKKJYJKPK43E2WKZ5LD4MEWXH3N54V46AZUYCJYA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.878724030 (504h18m4.749807075s) [noise=true , address=77.90.31.19:4854 , piece_id=UCSS6VRDOAD2C46W4FZ6JY2Y37H7N7H5R3COTXZVG3RMJLSKUUFA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.906686897 (504h18m4.777769932s) [noise=true , address=77.90.31.18:4821 , piece_id=VBTID3ZSFHPFRDO7LDMBHB4UGZ5TMXITVRGPFKGVDMGIMXORC3CA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.113831026 (504h18m4.984914061s) [noise=true , address=77.90.31.12:4597 , piece_id=JHVQ75DVU5EPWOJXNKS24424XANPKG4OM3V2BEZFNKQEGVCERKZA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.217449733 (504h18m5.088532778s) [noise=true , address=45.91.116.10:4092 , piece_id=4PC6UULS6ORGKBKLAU252LL3TEIPG33NBJQYPUHHZACUVTD6ADPA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.443177628 (504h18m5.314260663s) [noise=true , address=145.79.227.44:4205 , piece_id=XBCTB3HFCLOY3PWYANKNF3XTITDFL2M2GN436Y3WFEKCWBY7K2BQ, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.460351762 (504h18m5.331434807s) [noise=true , address=145.79.227.30:4777 , piece_id=VCYPI4657CWNMMNYGXEUIVCOTJY3GSEI3F7GKQ546QIEAWLE2ZEA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.495633474 (504h18m5.36671652s) [noise=true , address=145.79.227.34:4936 , piece_id=P6YGSLH57DM2W47VSRJVND3WVVDHTS4ODEBMFIJNHTZGDUSESSBA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.512258741 (504h18m5.383341786s) [noise=true , address=145.79.227.40:4083 , piece_id=QYVAADWLI7ZNMN6N5SAPA35CWS5Y6NZYKL7LLKWIFLMCYYKXH4UA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.622007582 (504h18m5.493090637s) [noise=true , address=145.79.227.33:4885 , piece_id=UXXEZZTTTHZJFLR2ZCWFA4SR2IDF2TMPOATWRXNMQDML77FZLS3A, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.828010718 (504h18m5.699093763s) [noise=true , address=45.91.116.13:4165 , piece_id=SMSJAQ27WZIXIBJHG2VB42XZXUEI7VGC5VRVOH7PXNDHUH5WETPA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.828033271 (504h18m5.699116316s) [seg_pos={0 0}, upload=442530]: Segment reached optimal threshold of 14 pieces. Dec 26 15:47:05.828240316 (504h18m5.699323361s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection Dec 26 15:47:05.828247699 (504h18m5.699330745s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828383899 (504h18m5.699466944s) [noise=true , address=77.90.31.11:4562 , piece_id=2RIE4646FNTIN6NFI6NJBI3VDDUFW2RQZ5LTZQVHBN6JPCISMWKA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 26 15:47:05.828390852 (504h18m5.699473897s) [noise=true , address=77.90.31.11:4562 , piece_id=2RIE4646FNTIN6NFI6NJBI3VDDUFW2RQZ5LTZQVHBN6JPCISMWKA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828580433 (504h18m5.699663499s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh): protocol: write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection; write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection Dec 26 15:47:05.828607355 (504h18m5.69969039s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828998109 (504h18m5.700081154s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XwcXkyKfaFNtGQpfU6npzHYAz2ySySmvdT9ya9HUa7u6dR2Zy): protocol: context canceled; context canceled Dec 26 15:47:05.829053253 (504h18m5.700136288s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.829080796 (504h18m5.700163841s) [seg_pos={0 0}, upload=442530]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:47:05.829185195 (504h18m5.70026824s) [upload=442530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:47:05.829190374 (504h18m5.700273419s) [upload=442530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:47:06.133692099 (504h18m6.004775144s) [upload=442530]: Done uploading segments... Dec 26 15:47:06.133710274 (504h18m6.004793319s) [upload=442530]: Upload finished. err: <nil> Duration: 45.807060709s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X....XX...] duration: 5.001097902s trace: https://jaeger.dev.storj.io/trace/469f1a4847e5645c FAIL (2025-12-26 15:49:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001127921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.X............XXX....XX...X......X..X..X.X......XX........] duration: 5.813162395s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:49:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.81319077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.X............XXX....XX...X......X..X..X.X......XX.......] duration: 5.828476618s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:49:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.8284937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX.XXXX.X..................X..........X.XX.X....X....X..X.] duration: 57.783670345s trace: https://jaeger.dev.storj.io/trace/5e07f788228f512 FAIL (2025-12-26 15:49:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (504h18m2.510656332s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.639766805 (504h18m2.51084984s) [noise=true , address=77.90.31.19:4854 , piece_id=UCSS6VRDOAD2C46W4FZ6JY2Y37H7N7H5R3COTXZVG3RMJLSKUUFA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.639948822 (504h18m2.511031857s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640103847 (504h18m2.511186892s) [noise=true , address=45.91.116.10:4092 , piece_id=4PC6UULS6ORGKBKLAU252LL3TEIPG33NBJQYPUHHZACUVTD6ADPA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640330458 (504h18m2.511413503s) [noise=true , address=193.149.164.143:4701 , piece_id=O6VGCDFTSGYTN6F7D72K6JOA667ZMFV3O6DRKCYXJEL6NJFNCFWQ, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640555597 (504h18m2.511638633s) [noise=true , address=145.79.227.30:4777 , piece_id=VCYPI4657CWNMMNYGXEUIVCOTJY3GSEI3F7GKQ546QIEAWLE2ZEA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640738696 (504h18m2.511821761s) [noise=true , address=77.90.31.18:4821 , piece_id=VBTID3ZSFHPFRDO7LDMBHB4UGZ5TMXITVRGPFKGVDMGIMXORC3CA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640904802 (504h18m2.511987837s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641146643 (504h18m2.512229678s) [noise=true , address=145.79.227.40:4083 , piece_id=QYVAADWLI7ZNMN6N5SAPA35CWS5Y6NZYKL7LLKWIFLMCYYKXH4UA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641438939 (504h18m2.512521984s) [noise=true , address=145.79.227.34:4936 , piece_id=P6YGSLH57DM2W47VSRJVND3WVVDHTS4ODEBMFIJNHTZGDUSESSBA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641601249 (504h18m2.512684284s) [noise=true , address=145.79.227.33:4885 , piece_id=UXXEZZTTTHZJFLR2ZCWFA4SR2IDF2TMPOATWRXNMQDML77FZLS3A, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641873547 (504h18m2.512956582s) [noise=true , address=45.91.116.13:4165 , piece_id=SMSJAQ27WZIXIBJHG2VB42XZXUEI7VGC5VRVOH7PXNDHUH5WETPA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642045344 (504h18m2.51312839s) [noise=true , address=77.90.31.15:4706 , piece_id=EQF53SQSFEXSYKKJYJKPK43E2WKZ5LD4MEWXH3N54V46AZUYCJYA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642234935 (504h18m2.51331798s) [noise=true , address=145.79.227.14:4850 , piece_id=HWBSZFDBWYAAGQTLXBW3ZNPMUAFC3U7KXARKCYQHPCC5E55AEZJA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642576606 (504h18m2.513659651s) [noise=true , address=77.90.31.12:4597 , piece_id=JHVQ75DVU5EPWOJXNKS24424XANPKG4OM3V2BEZFNKQEGVCERKZA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:03.955474533 (504h18m3.826557578s) [upload=442530]: Next returned nil segment Dec 26 15:47:03.955496284 (504h18m3.826579319s) [upload=442530]: Waiting for error group managing segments... Dec 26 15:47:04.292162345 (504h18m4.1632454s) [noise=true , address=193.149.164.143:4701 , piece_id=O6VGCDFTSGYTN6F7D72K6JOA667ZMFV3O6DRKCYXJEL6NJFNCFWQ, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.573847513 (504h18m4.444930558s) [noise=true , address=145.79.227.14:4850 , piece_id=HWBSZFDBWYAAGQTLXBW3ZNPMUAFC3U7KXARKCYQHPCC5E55AEZJA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.692327759 (504h18m4.563410804s) [noise=true , address=77.90.31.13:4656 , piece_id=3HFWM7ORA6VYY5XMGM5GHHQ5YAELNK6DLZUOQEQQ5WSL6KDGMOQA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.816015032 (504h18m4.687098067s) [noise=true , address=77.90.31.15:4706 , piece_id=EQF53SQSFEXSYKKJYJKPK43E2WKZ5LD4MEWXH3N54V46AZUYCJYA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.878724030 (504h18m4.749807075s) [noise=true , address=77.90.31.19:4854 , piece_id=UCSS6VRDOAD2C46W4FZ6JY2Y37H7N7H5R3COTXZVG3RMJLSKUUFA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.906686897 (504h18m4.777769932s) [noise=true , address=77.90.31.18:4821 , piece_id=VBTID3ZSFHPFRDO7LDMBHB4UGZ5TMXITVRGPFKGVDMGIMXORC3CA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.113831026 (504h18m4.984914061s) [noise=true , address=77.90.31.12:4597 , piece_id=JHVQ75DVU5EPWOJXNKS24424XANPKG4OM3V2BEZFNKQEGVCERKZA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.217449733 (504h18m5.088532778s) [noise=true , address=45.91.116.10:4092 , piece_id=4PC6UULS6ORGKBKLAU252LL3TEIPG33NBJQYPUHHZACUVTD6ADPA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.443177628 (504h18m5.314260663s) [noise=true , address=145.79.227.44:4205 , piece_id=XBCTB3HFCLOY3PWYANKNF3XTITDFL2M2GN436Y3WFEKCWBY7K2BQ, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.460351762 (504h18m5.331434807s) [noise=true , address=145.79.227.30:4777 , piece_id=VCYPI4657CWNMMNYGXEUIVCOTJY3GSEI3F7GKQ546QIEAWLE2ZEA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.495633474 (504h18m5.36671652s) [noise=true , address=145.79.227.34:4936 , piece_id=P6YGSLH57DM2W47VSRJVND3WVVDHTS4ODEBMFIJNHTZGDUSESSBA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.512258741 (504h18m5.383341786s) [noise=true , address=145.79.227.40:4083 , piece_id=QYVAADWLI7ZNMN6N5SAPA35CWS5Y6NZYKL7LLKWIFLMCYYKXH4UA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.622007582 (504h18m5.493090637s) [noise=true , address=145.79.227.33:4885 , piece_id=UXXEZZTTTHZJFLR2ZCWFA4SR2IDF2TMPOATWRXNMQDML77FZLS3A, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.828010718 (504h18m5.699093763s) [noise=true , address=45.91.116.13:4165 , piece_id=SMSJAQ27WZIXIBJHG2VB42XZXUEI7VGC5VRVOH7PXNDHUH5WETPA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.828033271 (504h18m5.699116316s) [seg_pos={0 0}, upload=442530]: Segment reached optimal threshold of 14 pieces. Dec 26 15:47:05.828240316 (504h18m5.699323361s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection Dec 26 15:47:05.828247699 (504h18m5.699330745s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828383899 (504h18m5.699466944s) [noise=true , address=77.90.31.11:4562 , piece_id=2RIE4646FNTIN6NFI6NJBI3VDDUFW2RQZ5LTZQVHBN6JPCISMWKA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 26 15:47:05.828390852 (504h18m5.699473897s) [noise=true , address=77.90.31.11:4562 , piece_id=2RIE4646FNTIN6NFI6NJBI3VDDUFW2RQZ5LTZQVHBN6JPCISMWKA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828580433 (504h18m5.699663499s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh): protocol: write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection; write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection Dec 26 15:47:05.828607355 (504h18m5.69969039s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828998109 (504h18m5.700081154s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XwcXkyKfaFNtGQpfU6npzHYAz2ySySmvdT9ya9HUa7u6dR2Zy): protocol: context canceled; context canceled Dec 26 15:47:05.829053253 (504h18m5.700136288s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.829080796 (504h18m5.700163841s) [seg_pos={0 0}, upload=442530]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:47:05.829185195 (504h18m5.70026824s) [upload=442530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:47:05.829190374 (504h18m5.700273419s) [upload=442530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:47:06.133692099 (504h18m6.004775144s) [upload=442530]: Done uploading segments... Dec 26 15:47:06.133710274 (504h18m6.004793319s) [upload=442530]: Upload finished. err: <nil> Duration: 57.783700973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X....XX..] duration: 3.41668288s trace: https://jaeger.dev.storj.io/trace/469f1a4847e5645c FAIL (2025-12-26 15:49:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.416715974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X.X............XXX....XX...X......X..X..X.X......XX......] duration: 6.390366483s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:48:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 6.3903913s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X....XX.] duration: 5.00106481s trace: https://jaeger.dev.storj.io/trace/469f1a4847e5645c FAIL (2025-12-26 15:48:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001089727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [...XX.XXXX.X..................X..........X.XX.X....X....X..X] duration: 1m0.002914332s trace: https://jaeger.dev.storj.io/trace/5e07f788228f512 FAIL (2025-12-26 15:48:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (504h18m2.510656332s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.639766805 (504h18m2.51084984s) [noise=true , address=77.90.31.19:4854 , piece_id=UCSS6VRDOAD2C46W4FZ6JY2Y37H7N7H5R3COTXZVG3RMJLSKUUFA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.639948822 (504h18m2.511031857s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640103847 (504h18m2.511186892s) [noise=true , address=45.91.116.10:4092 , piece_id=4PC6UULS6ORGKBKLAU252LL3TEIPG33NBJQYPUHHZACUVTD6ADPA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640330458 (504h18m2.511413503s) [noise=true , address=193.149.164.143:4701 , piece_id=O6VGCDFTSGYTN6F7D72K6JOA667ZMFV3O6DRKCYXJEL6NJFNCFWQ, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640555597 (504h18m2.511638633s) [noise=true , address=145.79.227.30:4777 , piece_id=VCYPI4657CWNMMNYGXEUIVCOTJY3GSEI3F7GKQ546QIEAWLE2ZEA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640738696 (504h18m2.511821761s) [noise=true , address=77.90.31.18:4821 , piece_id=VBTID3ZSFHPFRDO7LDMBHB4UGZ5TMXITVRGPFKGVDMGIMXORC3CA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.640904802 (504h18m2.511987837s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641146643 (504h18m2.512229678s) [noise=true , address=145.79.227.40:4083 , piece_id=QYVAADWLI7ZNMN6N5SAPA35CWS5Y6NZYKL7LLKWIFLMCYYKXH4UA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641438939 (504h18m2.512521984s) [noise=true , address=145.79.227.34:4936 , piece_id=P6YGSLH57DM2W47VSRJVND3WVVDHTS4ODEBMFIJNHTZGDUSESSBA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641601249 (504h18m2.512684284s) [noise=true , address=145.79.227.33:4885 , piece_id=UXXEZZTTTHZJFLR2ZCWFA4SR2IDF2TMPOATWRXNMQDML77FZLS3A, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.641873547 (504h18m2.512956582s) [noise=true , address=45.91.116.13:4165 , piece_id=SMSJAQ27WZIXIBJHG2VB42XZXUEI7VGC5VRVOH7PXNDHUH5WETPA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642045344 (504h18m2.51312839s) [noise=true , address=77.90.31.15:4706 , piece_id=EQF53SQSFEXSYKKJYJKPK43E2WKZ5LD4MEWXH3N54V46AZUYCJYA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642234935 (504h18m2.51331798s) [noise=true , address=145.79.227.14:4850 , piece_id=HWBSZFDBWYAAGQTLXBW3ZNPMUAFC3U7KXARKCYQHPCC5E55AEZJA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:02.642576606 (504h18m2.513659651s) [noise=true , address=77.90.31.12:4597 , piece_id=JHVQ75DVU5EPWOJXNKS24424XANPKG4OM3V2BEZFNKQEGVCERKZA, seg_pos={0 0}, upload=442530]: Uploading piece... Dec 26 15:47:03.955474533 (504h18m3.826557578s) [upload=442530]: Next returned nil segment Dec 26 15:47:03.955496284 (504h18m3.826579319s) [upload=442530]: Waiting for error group managing segments... Dec 26 15:47:04.292162345 (504h18m4.1632454s) [noise=true , address=193.149.164.143:4701 , piece_id=O6VGCDFTSGYTN6F7D72K6JOA667ZMFV3O6DRKCYXJEL6NJFNCFWQ, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.573847513 (504h18m4.444930558s) [noise=true , address=145.79.227.14:4850 , piece_id=HWBSZFDBWYAAGQTLXBW3ZNPMUAFC3U7KXARKCYQHPCC5E55AEZJA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.692327759 (504h18m4.563410804s) [noise=true , address=77.90.31.13:4656 , piece_id=3HFWM7ORA6VYY5XMGM5GHHQ5YAELNK6DLZUOQEQQ5WSL6KDGMOQA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.816015032 (504h18m4.687098067s) [noise=true , address=77.90.31.15:4706 , piece_id=EQF53SQSFEXSYKKJYJKPK43E2WKZ5LD4MEWXH3N54V46AZUYCJYA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.878724030 (504h18m4.749807075s) [noise=true , address=77.90.31.19:4854 , piece_id=UCSS6VRDOAD2C46W4FZ6JY2Y37H7N7H5R3COTXZVG3RMJLSKUUFA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:04.906686897 (504h18m4.777769932s) [noise=true , address=77.90.31.18:4821 , piece_id=VBTID3ZSFHPFRDO7LDMBHB4UGZ5TMXITVRGPFKGVDMGIMXORC3CA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.113831026 (504h18m4.984914061s) [noise=true , address=77.90.31.12:4597 , piece_id=JHVQ75DVU5EPWOJXNKS24424XANPKG4OM3V2BEZFNKQEGVCERKZA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.217449733 (504h18m5.088532778s) [noise=true , address=45.91.116.10:4092 , piece_id=4PC6UULS6ORGKBKLAU252LL3TEIPG33NBJQYPUHHZACUVTD6ADPA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.443177628 (504h18m5.314260663s) [noise=true , address=145.79.227.44:4205 , piece_id=XBCTB3HFCLOY3PWYANKNF3XTITDFL2M2GN436Y3WFEKCWBY7K2BQ, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.460351762 (504h18m5.331434807s) [noise=true , address=145.79.227.30:4777 , piece_id=VCYPI4657CWNMMNYGXEUIVCOTJY3GSEI3F7GKQ546QIEAWLE2ZEA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.495633474 (504h18m5.36671652s) [noise=true , address=145.79.227.34:4936 , piece_id=P6YGSLH57DM2W47VSRJVND3WVVDHTS4ODEBMFIJNHTZGDUSESSBA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.512258741 (504h18m5.383341786s) [noise=true , address=145.79.227.40:4083 , piece_id=QYVAADWLI7ZNMN6N5SAPA35CWS5Y6NZYKL7LLKWIFLMCYYKXH4UA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.622007582 (504h18m5.493090637s) [noise=true , address=145.79.227.33:4885 , piece_id=UXXEZZTTTHZJFLR2ZCWFA4SR2IDF2TMPOATWRXNMQDML77FZLS3A, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.828010718 (504h18m5.699093763s) [noise=true , address=45.91.116.13:4165 , piece_id=SMSJAQ27WZIXIBJHG2VB42XZXUEI7VGC5VRVOH7PXNDHUH5WETPA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: <nil> Dec 26 15:47:05.828033271 (504h18m5.699116316s) [seg_pos={0 0}, upload=442530]: Segment reached optimal threshold of 14 pieces. Dec 26 15:47:05.828240316 (504h18m5.699323361s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47978->64.49.13.169:4705: use of closed network connection Dec 26 15:47:05.828247699 (504h18m5.699330745s) [noise=true , address=64.49.13.169:4705 , piece_id=ZFLZYFX5Z62N4C5OWSJVYKNY7IQHDCDZOIEII53ERULUEL5YBKZA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828383899 (504h18m5.699466944s) [noise=true , address=77.90.31.11:4562 , piece_id=2RIE4646FNTIN6NFI6NJBI3VDDUFW2RQZ5LTZQVHBN6JPCISMWKA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 26 15:47:05.828390852 (504h18m5.699473897s) [noise=true , address=77.90.31.11:4562 , piece_id=2RIE4646FNTIN6NFI6NJBI3VDDUFW2RQZ5LTZQVHBN6JPCISMWKA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828580433 (504h18m5.699663499s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh): protocol: write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection; write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59464->77.90.31.14:4673: use of closed network connection Dec 26 15:47:05.828607355 (504h18m5.69969039s) [noise=true , address=77.90.31.14:4673 , piece_id=6NSB7EZ6Z3D2XKWK6P3BEEIKJSPFMGNJH2VZYD6ARR3TRAXMAY4A, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.828998109 (504h18m5.700081154s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XwcXkyKfaFNtGQpfU6npzHYAz2ySySmvdT9ya9HUa7u6dR2Zy): protocol: context canceled; context canceled Dec 26 15:47:05.829053253 (504h18m5.700136288s) [noise=true , address=64.49.13.27:4959 , piece_id=LRWC2BYPK5O27H6A44UTYX6IKT6GA53VRR3T6D7JE4ZCQG3F5IGA, seg_pos={0 0}, upload=442530]: Long tail cancellation due to optimal threshold Dec 26 15:47:05.829080796 (504h18m5.700163841s) [seg_pos={0 0}, upload=442530]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:47:05.829185195 (504h18m5.70026824s) [upload=442530]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:47:05.829190374 (504h18m5.700273419s) [upload=442530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:47:06.133692099 (504h18m6.004775144s) [upload=442530]: Done uploading segments... Dec 26 15:47:06.133710274 (504h18m6.004793319s) [upload=442530]: Upload finished. err: <nil> Duration: 1m0.002965299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X.X............XXX....XX...X......X..X..X.X......XX.....] duration: 5.727681946s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:48:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.727729476s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....X.X............XXX....XX...X......X..X..X.X......XX....] duration: 5.830248973s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:47:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.830277657s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [XX..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X....XX] duration: 5.001421639s trace: https://jaeger.dev.storj.io/trace/469f1a4847e5645c FAIL (2025-12-26 15:47:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001443282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.....X.X............XXX....XX...X......X..X..X.X......XX...] duration: 6.295271635s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:47:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 6.295299189s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...XX.XXXX.X..................X..........X.XX.X....X....X..] duration: 38.342010489s trace: https://jaeger.dev.storj.io/trace/4cd8fa1e4f5e1dc8 FAIL (2025-12-26 15:46:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: 5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA) context deadline exceeded; piecestore: (Node ID: 12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, Piece ID: 5MOGOYCIZDUF3RBK7KBVNJDKS5JONLHBG6D2UVZUDL45MWN2P4TA) context deadline exceeded; piecestore: (Node ID: 12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, Piece ID: Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ) context deadline exceeded ======LOGS====== 629 (504h15m24.782960674s) [noise=true , address=145.79.227.52:4449 , piece_id=TZBWOOBMMSPSNEWKBLAAHDRSKLCBDKRZQC3KXCCRH7QCHQ7QJUYA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912000853 (504h15m24.783083918s) [noise=true , address=145.79.227.34:4923 , piece_id=PPAPN6DNQG5RBVMEW7PJWW37KULMSHFBZ24AGMMDQ7IPUWZGETAQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912414501 (504h15m24.783497546s) [noise=true , address=77.90.31.17:4799 , piece_id=SQISFPF6L5V6JW6BP2COCZF7Z4X3BPUE2D3SWTHH3EKNJM5WZFXQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912565018 (504h15m24.783648063s) [noise=true , address=145.79.227.16:4914 , piece_id=EBYZPSEJGMOPUKXTDJ3PWIYBPWVVWKQB7TQ2EJM3CVOIZH66GSQA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912793312 (504h15m24.783876357s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.913155432 (504h15m24.784238477s) [noise=true , address=77.90.31.21:4895 , piece_id=5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.913447237 (504h15m24.784530272s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.914073700 (504h15m24.785156755s) [noise=true , address=145.79.227.40:4055 , piece_id=5LHHIMTDTOAIMBJBGQJHTVYK4XHV2CDIFKCQJ464LGZF56LVHGTQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.914654306 (504h15m24.785737351s) [noise=true , address=64.49.13.27:4961 , piece_id=SWD24N7WNLCAAWI7PJYLSLH6MOXXO44RSJNMNFHWUJ3NFPIUP6YA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.915261752 (504h15m24.786344798s) [noise=true , address=45.91.116.10:4092 , piece_id=LFKFNJRC7VZM3YKJRESLYNM746DG2QQDBR2CMKOJWQKGWWVXE62Q, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.915584346 (504h15m24.786667392s) [noise=true , address=77.90.31.15:4708 , piece_id=4TZTHMBJ7JLAKD4PTWBG5OBS2SHM7ARDW4MX2WGYSB5BOOAVSDPA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916034153 (504h15m24.787117188s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916473640 (504h15m24.787556675s) [noise=true , address=77.90.31.18:4836 , piece_id=6K7KXGECA6OM6UPZ5QYK246WOHQTB3M4NSXNHJX3BYSYLIZNVPSQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916980105 (504h15m24.78806315s) [noise=true , address=77.90.31.16:4751 , piece_id=Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.917377612 (504h15m24.788460657s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:26.223735515 (504h15m26.09481856s) [upload=442494]: Next returned nil segment Dec 26 15:44:26.223748189 (504h15m26.094831224s) [upload=442494]: Waiting for error group managing segments... Dec 26 15:44:26.592105696 (504h15m26.463188742s) [noise=true , address=64.49.13.137:4499 , piece_id=T5CKU2H2XUCA4OWMOM7RHK7G32EEJTCHOENTBDXZEH56A7QHCSCA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.791746046 (504h15m26.662829091s) [noise=true , address=145.79.227.16:4914 , piece_id=EBYZPSEJGMOPUKXTDJ3PWIYBPWVVWKQB7TQ2EJM3CVOIZH66GSQA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.866532121 (504h15m26.737615166s) [noise=true , address=64.49.13.27:4961 , piece_id=SWD24N7WNLCAAWI7PJYLSLH6MOXXO44RSJNMNFHWUJ3NFPIUP6YA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.945299659 (504h15m26.816382694s) [noise=true , address=145.79.227.52:4449 , piece_id=TZBWOOBMMSPSNEWKBLAAHDRSKLCBDKRZQC3KXCCRH7QCHQ7QJUYA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.097735153 (504h15m26.968818188s) [noise=true , address=77.90.31.13:4625 , piece_id=KH3MOO4QFLK237MTUUG7DJZOJT6P3UQFHR4PKVL4SXHU5N32BDVA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.220855005 (504h15m27.09193804s) [noise=true , address=77.90.31.21:4895 , piece_id=5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.339482502 (504h15m27.210565547s) [noise=true , address=77.90.31.17:4799 , piece_id=SQISFPF6L5V6JW6BP2COCZF7Z4X3BPUE2D3SWTHH3EKNJM5WZFXQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.637331558 (504h15m27.508414603s) [noise=true , address=77.90.31.12:4621 , piece_id=5MOGOYCIZDUF3RBK7KBVNJDKS5JONLHBG6D2UVZUDL45MWN2P4TA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.708229921 (504h15m27.579312966s) [noise=true , address=145.79.227.40:4055 , piece_id=5LHHIMTDTOAIMBJBGQJHTVYK4XHV2CDIFKCQJ464LGZF56LVHGTQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.739740644 (504h15m27.610823689s) [noise=true , address=145.79.227.34:4923 , piece_id=PPAPN6DNQG5RBVMEW7PJWW37KULMSHFBZ24AGMMDQ7IPUWZGETAQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.746887031 (504h15m27.617970076s) [noise=true , address=77.90.31.15:4708 , piece_id=4TZTHMBJ7JLAKD4PTWBG5OBS2SHM7ARDW4MX2WGYSB5BOOAVSDPA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.979849332 (504h15m27.850932367s) [noise=true , address=77.90.31.18:4836 , piece_id=6K7KXGECA6OM6UPZ5QYK246WOHQTB3M4NSXNHJX3BYSYLIZNVPSQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.390591527 (504h15m28.261674562s) [noise=true , address=77.90.31.16:4751 , piece_id=Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.569880514 (504h15m28.440963559s) [noise=true , address=45.91.116.10:4092 , piece_id=LFKFNJRC7VZM3YKJRESLYNM746DG2QQDBR2CMKOJWQKGWWVXE62Q, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.569901214 (504h15m28.440984249s) [seg_pos={0 0}, upload=442494]: Segment reached optimal threshold of 14 pieces. Dec 26 15:44:28.570240661 (504h15m28.441323706s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 26 15:44:28.570256761 (504h15m28.441339796s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570262151 (504h15m28.441345186s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection Dec 26 15:44:28.570271910 (504h15m28.441354945s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570537195 (504h15m28.44162023s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: rpc: dial tcp 45.91.116.4:4948: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:44:28.570544318 (504h15m28.441627353s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570740913 (504h15m28.441823958s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: read tcp 188.166.225.170:59608->45.91.116.6:4999: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:44:28.570753126 (504h15m28.441836161s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570780768 (504h15m28.441863814s) [seg_pos={0 0}, upload=442494]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:44:28.570883895 (504h15m28.44196693s) [upload=442494]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:44:28.570907761 (504h15m28.441990816s) [upload=442494]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:44:28.878863402 (504h15m28.749946477s) [upload=442494]: Done uploading segments... Dec 26 15:44:28.878894812 (504h15m28.749977887s) [upload=442494]: Upload finished. err: <nil> Duration: 38.34202678s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [XXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X....X] duration: 5.001079307s trace: https://jaeger.dev.storj.io/trace/584d072c20d409d8 FAIL (2025-12-26 15:46:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001114896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.....X.X............XXX....XX...X......X..X..X.X......XX..] duration: 5.798977725s trace: https://jaeger.dev.storj.io/trace/4f21aec5b8e4a8fc FAIL (2025-12-26 15:46:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:44:59.986754845 (504h15m59.85783788s) [upload=442504]: Starting upload Dec 26 15:44:59.986842182 (504h15m59.857925257s) [upload=442504]: Done starting upload Dec 26 15:44:59.986905753 (504h15m59.857988798s) [upload=442504]: Uploading segments... Dec 26 15:44:59.987044948 (504h15m59.858127983s) [upload=442504]: Got next segment. Inline: false Dec 26 15:44:59.987086838 (504h15m59.858169883s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:44:59.987091236 (504h15m59.858174271s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:45:00.234035842 (504h16m0.105118877s) [seg_pos={0 0}, upload=442504]: Begin upload segment... Dec 26 15:45:00.234088121 (504h16m0.105171156s) [seg_pos={0 0}, upload=442504]: Done begin upload segment. Dec 26 15:45:00.234167813 (504h16m0.105250858s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234202579 (504h16m0.105285614s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234313430 (504h16m0.105396475s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.234841576 (504h16m0.105924631s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235445556 (504h16m0.106528601s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.235987679 (504h16m0.107070724s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236593953 (504h16m0.107676998s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.236891520 (504h16m0.107974565s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237012199 (504h16m0.108095244s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.237544873 (504h16m0.108627918s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.239426416 (504h16m0.110509451s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.245043022 (504h16m0.116126067s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.249426338 (504h16m0.120509383s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.254857911 (504h16m0.125940946s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.257845931 (504h16m0.128929066s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:00.259400550 (504h16m0.130483585s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Uploading piece... Dec 26 15:45:01.996751991 (504h16m1.867835046s) [upload=442504]: Next returned nil segment Dec 26 15:45:01.996772961 (504h16m1.867856006s) [upload=442504]: Waiting for error group managing segments... Dec 26 15:45:02.309232458 (504h16m2.180315504s) [noise=true , address=145.79.227.49:4339 , piece_id=M6S3Q37HADGA2Z7GHH57ZIS43DPPITU7GTRWYTVCHD6WJCB64KSQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.320630402 (504h16m2.191713447s) [noise=true , address=145.79.227.39:4014 , piece_id=HZQVQHQY7FOAHYHJ6RWWM57KT2QJEBYKENFIO7NCS2Q7F2GK62DQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.326234484 (504h16m2.197317529s) [noise=true , address=145.79.227.38:4979 , piece_id=5DBSB7TBL6FMELWQPSTTGOC72M2EQMP6VUYD6VY7V7DP3KGA2BZQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.327123977 (504h16m2.198207022s) [noise=true , address=145.79.227.7:4293 , piece_id=JHD4SA6NQV4LWPVVAD2GHO4OBA25X6ENZNBWKAJMNKDXZAM7B5JA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.334884063 (504h16m2.205967118s) [noise=true , address=145.79.227.2:4115 , piece_id=3CIPB22T4QSCSRPDDB5JGF77QOGXEOHQKYOZ4MK37M7KSUF6JBFA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.335473676 (504h16m2.206556731s) [noise=true , address=145.79.227.51:4410 , piece_id=5RR7VEAGR4OTPGFSIQYTPI6KSOPUUPGJRQIUPFIB7XR6XKKNZXVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.341240337 (504h16m2.212323382s) [noise=true , address=145.79.227.16:4912 , piece_id=XISZHX6JMQUBNRTUAHSD2HT4EWFYKOVGOBHN3S5GNDJDK3MVYOQQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.358609181 (504h16m2.229692226s) [noise=true , address=145.79.227.29:4943 , piece_id=4SEYGLUYXZARCRTKMLDVU43K25P4VU6VOJZVCR7SXT5E2QLSGL5Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.660913321 (504h16m2.531996376s) [noise=true , address=77.90.31.17:4773 , piece_id=BQ3NNRP4F6NVVMYTKHMETRYPMEAODKBTPO2242NUREBPPAAXW62Q, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.680245232 (504h16m2.551328277s) [noise=true , address=45.91.116.5:4300 , piece_id=HRLL2OWPLA6F3Q7QOG5QY7MTWYCYAAAYCEPBQLPJ4O2QVTNQSDJQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.704956476 (504h16m2.576039521s) [noise=true , address=45.91.116.8:4047 , piece_id=HKHIKBEMRNY3YDBKUVXPQJ6KAUVCN3P7YF6P327NJ673XHVKBYVQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:02.754023919 (504h16m2.625106975s) [noise=true , address=45.91.116.10:4082 , piece_id=OFRCKNBMJIBX465HBANBYV765RJ4B3MFXEM5H7ZELCFWO7UVKCXQ, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235256069 (504h16m3.106339114s) [noise=true , address=45.91.116.6:4974 , piece_id=DQFUMGZTBLLP7DITUXKAQG2YRXRG45KVGORJKHDZ7MW6RRFBYTKA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: <nil> Dec 26 15:45:03.235284574 (504h16m3.106367619s) [seg_pos={0 0}, upload=442504]: Segment reached optimal threshold of 13 pieces. Dec 26 15:45:03.235588241 (504h16m3.106671296s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39874->45.91.116.9:4261: use of closed network connection Dec 26 15:45:03.235600345 (504h16m3.10668339s) [noise=true , address=45.91.116.9:4261 , piece_id=CYOO6ZSWXAJA54ML2TGI6GS323IHM2WOE6XP5YK2XNH4HYYVEIWA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236080580 (504h16m3.107163625s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236089837 (504h16m3.107172892s) [noise=true , address=77.90.31.16:4736 , piece_id=PUJS6OFP6H4EMPPTAY53YHL7DCPPYHKVQ7ODCXXKLVFOIS7KSOYA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236240914 (504h16m3.107323959s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 15:45:03.236247847 (504h16m3.107330892s) [noise=true , address=77.90.31.10:4517 , piece_id=YOWDFYEWNIPEN4HWJFV7WQNVARQDT4PVP6K23OEH5QA7Q5MOVVRA, seg_pos={0 0}, upload=442504]: Long tail cancellation due to optimal threshold Dec 26 15:45:03.236265992 (504h16m3.107349027s) [seg_pos={0 0}, upload=442504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:45:03.236358198 (504h16m3.107441253s) [upload=442504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:45:03.236368347 (504h16m3.107451392s) [upload=442504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:45:03.552166912 (504h16m3.423249958s) [upload=442504]: Done uploading segments... Dec 26 15:45:03.552193834 (504h16m3.423276879s) [upload=442504]: Upload finished. err: <nil> Duration: 5.799028152s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [XXXX..X...X.X.X........XXX..X.XX...X.....X.X...XX.XX...X....] duration: 5.000496918s trace: https://jaeger.dev.storj.io/trace/3872e6d9920cd189 FAIL (2025-12-26 15:46:21 UTC) orbiter warning: context deadline exceeded Duration: 5.000526114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X...XX.XXXX.X..................X..........X.XX.X....X....X.] duration: 50.220693646s trace: https://jaeger.dev.storj.io/trace/4cd8fa1e4f5e1dc8 FAIL (2025-12-26 15:46:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: 5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA) context deadline exceeded; piecestore: (Node ID: 12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, Piece ID: 5MOGOYCIZDUF3RBK7KBVNJDKS5JONLHBG6D2UVZUDL45MWN2P4TA) context deadline exceeded; piecestore: (Node ID: 12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, Piece ID: Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ) context deadline exceeded ======LOGS====== 629 (504h15m24.782960674s) [noise=true , address=145.79.227.52:4449 , piece_id=TZBWOOBMMSPSNEWKBLAAHDRSKLCBDKRZQC3KXCCRH7QCHQ7QJUYA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912000853 (504h15m24.783083918s) [noise=true , address=145.79.227.34:4923 , piece_id=PPAPN6DNQG5RBVMEW7PJWW37KULMSHFBZ24AGMMDQ7IPUWZGETAQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912414501 (504h15m24.783497546s) [noise=true , address=77.90.31.17:4799 , piece_id=SQISFPF6L5V6JW6BP2COCZF7Z4X3BPUE2D3SWTHH3EKNJM5WZFXQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912565018 (504h15m24.783648063s) [noise=true , address=145.79.227.16:4914 , piece_id=EBYZPSEJGMOPUKXTDJ3PWIYBPWVVWKQB7TQ2EJM3CVOIZH66GSQA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.912793312 (504h15m24.783876357s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.913155432 (504h15m24.784238477s) [noise=true , address=77.90.31.21:4895 , piece_id=5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.913447237 (504h15m24.784530272s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.914073700 (504h15m24.785156755s) [noise=true , address=145.79.227.40:4055 , piece_id=5LHHIMTDTOAIMBJBGQJHTVYK4XHV2CDIFKCQJ464LGZF56LVHGTQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.914654306 (504h15m24.785737351s) [noise=true , address=64.49.13.27:4961 , piece_id=SWD24N7WNLCAAWI7PJYLSLH6MOXXO44RSJNMNFHWUJ3NFPIUP6YA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.915261752 (504h15m24.786344798s) [noise=true , address=45.91.116.10:4092 , piece_id=LFKFNJRC7VZM3YKJRESLYNM746DG2QQDBR2CMKOJWQKGWWVXE62Q, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.915584346 (504h15m24.786667392s) [noise=true , address=77.90.31.15:4708 , piece_id=4TZTHMBJ7JLAKD4PTWBG5OBS2SHM7ARDW4MX2WGYSB5BOOAVSDPA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916034153 (504h15m24.787117188s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916473640 (504h15m24.787556675s) [noise=true , address=77.90.31.18:4836 , piece_id=6K7KXGECA6OM6UPZ5QYK246WOHQTB3M4NSXNHJX3BYSYLIZNVPSQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.916980105 (504h15m24.78806315s) [noise=true , address=77.90.31.16:4751 , piece_id=Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:24.917377612 (504h15m24.788460657s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Uploading piece... Dec 26 15:44:26.223735515 (504h15m26.09481856s) [upload=442494]: Next returned nil segment Dec 26 15:44:26.223748189 (504h15m26.094831224s) [upload=442494]: Waiting for error group managing segments... Dec 26 15:44:26.592105696 (504h15m26.463188742s) [noise=true , address=64.49.13.137:4499 , piece_id=T5CKU2H2XUCA4OWMOM7RHK7G32EEJTCHOENTBDXZEH56A7QHCSCA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.791746046 (504h15m26.662829091s) [noise=true , address=145.79.227.16:4914 , piece_id=EBYZPSEJGMOPUKXTDJ3PWIYBPWVVWKQB7TQ2EJM3CVOIZH66GSQA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.866532121 (504h15m26.737615166s) [noise=true , address=64.49.13.27:4961 , piece_id=SWD24N7WNLCAAWI7PJYLSLH6MOXXO44RSJNMNFHWUJ3NFPIUP6YA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:26.945299659 (504h15m26.816382694s) [noise=true , address=145.79.227.52:4449 , piece_id=TZBWOOBMMSPSNEWKBLAAHDRSKLCBDKRZQC3KXCCRH7QCHQ7QJUYA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.097735153 (504h15m26.968818188s) [noise=true , address=77.90.31.13:4625 , piece_id=KH3MOO4QFLK237MTUUG7DJZOJT6P3UQFHR4PKVL4SXHU5N32BDVA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.220855005 (504h15m27.09193804s) [noise=true , address=77.90.31.21:4895 , piece_id=5BBEPXKDBSNJSIU7M3KTS75OM7VJU4C7BF3RLAO3RBR6HRZLK2BA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.339482502 (504h15m27.210565547s) [noise=true , address=77.90.31.17:4799 , piece_id=SQISFPF6L5V6JW6BP2COCZF7Z4X3BPUE2D3SWTHH3EKNJM5WZFXQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.637331558 (504h15m27.508414603s) [noise=true , address=77.90.31.12:4621 , piece_id=5MOGOYCIZDUF3RBK7KBVNJDKS5JONLHBG6D2UVZUDL45MWN2P4TA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.708229921 (504h15m27.579312966s) [noise=true , address=145.79.227.40:4055 , piece_id=5LHHIMTDTOAIMBJBGQJHTVYK4XHV2CDIFKCQJ464LGZF56LVHGTQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.739740644 (504h15m27.610823689s) [noise=true , address=145.79.227.34:4923 , piece_id=PPAPN6DNQG5RBVMEW7PJWW37KULMSHFBZ24AGMMDQ7IPUWZGETAQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.746887031 (504h15m27.617970076s) [noise=true , address=77.90.31.15:4708 , piece_id=4TZTHMBJ7JLAKD4PTWBG5OBS2SHM7ARDW4MX2WGYSB5BOOAVSDPA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:27.979849332 (504h15m27.850932367s) [noise=true , address=77.90.31.18:4836 , piece_id=6K7KXGECA6OM6UPZ5QYK246WOHQTB3M4NSXNHJX3BYSYLIZNVPSQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.390591527 (504h15m28.261674562s) [noise=true , address=77.90.31.16:4751 , piece_id=Y5MR6RBQABTNUGQVHTTTFWDUHZKSH4QPCKWJDN2RBCB4ITKO46OQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.569880514 (504h15m28.440963559s) [noise=true , address=45.91.116.10:4092 , piece_id=LFKFNJRC7VZM3YKJRESLYNM746DG2QQDBR2CMKOJWQKGWWVXE62Q, seg_pos={0 0}, upload=442494]: Done uploading piece. err: <nil> Dec 26 15:44:28.569901214 (504h15m28.440984249s) [seg_pos={0 0}, upload=442494]: Segment reached optimal threshold of 14 pieces. Dec 26 15:44:28.570240661 (504h15m28.441323706s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 26 15:44:28.570256761 (504h15m28.441339796s) [noise=true , address=145.79.227.30:4766 , piece_id=SPPBBTVJX7HDY7KJ4QREVQP6B27TGWKKYR2I3WVELRJNN4PYPDPQ, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570262151 (504h15m28.441345186s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53764->64.49.13.169:4708: use of closed network connection Dec 26 15:44:28.570271910 (504h15m28.441354945s) [noise=true , address=64.49.13.169:4708 , piece_id=AMB6F3HPRCVUTWAM2U4O4YT7F4KIRPHIWFIXVYBZ2F5EZQQXDLNA, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570537195 (504h15m28.44162023s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: rpc: dial tcp 45.91.116.4:4948: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:44:28.570544318 (504h15m28.441627353s) [noise=true , address=45.91.116.4:4948 , piece_id=E2UNN3EB7NTLESRTFCQNXD54QN4SF2JRVFD5I6M7BCDFD6H6O7JA, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570740913 (504h15m28.441823958s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: read tcp 188.166.225.170:59608->45.91.116.6:4999: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:44:28.570753126 (504h15m28.441836161s) [noise=true , address=45.91.116.6:4999 , piece_id=T36VED2ASUTMZMFR5S3FHJ5CWIZPXZU7MXECVYH5X5EJ2ENURDUQ, seg_pos={0 0}, upload=442494]: Long tail cancellation due to optimal threshold Dec 26 15:44:28.570780768 (504h15m28.441863814s) [seg_pos={0 0}, upload=442494]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:44:28.570883895 (504h15m28.44196693s) [upload=442494]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:44:28.570907761 (504h15m28.441990816s) [upload=442494]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:44:28.878863402 (504h15m28.749946477s) [upload=442494]: Done uploading segments... Dec 26 15:44:28.878894812 (504h15m28.749977887s) [upload=442494]: Upload finished. err: <nil> Duration: 50.220715055s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)