Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.175359979s trace: https://jaeger.dev.storj.io/trace/4e6019303a470cb6 FAIL (2025-12-08 11:42:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.175378484s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X...............X..................] duration: 3.468715027s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:42:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.468745905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 15m8.168286798s trace: https://jaeger.dev.storj.io/trace/4e6019303a470cb6 FAIL (2025-12-08 11:42:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 15m8.168311415s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X................................X.........................] duration: 7.845802366s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:42:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.845833857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.891202724s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:41:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.891236527s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X................................X........................] duration: 7.071811698s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:41:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.071831295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X................................X.......................] duration: 8.984432072s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:41:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.98447849s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X...............X.................] duration: 3.512159672s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:41:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.512185431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 3.051170625s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:40:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.051196716s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X................................X......................] duration: 7.290838913s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:40:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.290875934s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X...............X................] duration: 3.500351366s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:40:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.500379931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X................................X.....................] duration: 10.19355602s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:39:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 10.193611135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.951064573s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:39:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.951103567s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X................................X....................] duration: 8.090666226s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:39:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.090700631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X...............X...............] duration: 3.492668476s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:39:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492717691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.001837415s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:39:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.00187636s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X................................X...................] duration: 7.061857582s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:38:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.061886477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.942811982s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:38:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.94284729s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X...............X..............] duration: 3.493913039s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:38:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.493944738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X................................X..................] duration: 6.539384831s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:37:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.539432793s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X...............X.............] duration: 3.521553989s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:37:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.521581843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.959743184s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:37:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.959774304s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X...............X............] duration: 3.408270876s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:37:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40830394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X................................X.................] duration: 7.928166643s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:36:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.928215856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.507471433s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:36:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.5075015s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X...............X...........] duration: 3.5205497s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:36:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520585989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X................................X................] duration: 7.238102304s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:36:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.238152079s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X...............X..........] duration: 3.464277455s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:36:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464301481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.925171597s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:35:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.925238924s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.059503316s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:35:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.059529126s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X................................X...............] duration: 7.158520442s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:35:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.158555648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X................................X..............] duration: 10.271788367s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:35:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 10.27181671s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X...............X.........] duration: 3.456200309s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:35:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.456232188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.387184509s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:34:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.387220538s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X...............X........] duration: 3.544040372s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:34:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.54407126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X................................X.............] duration: 13.768079542s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:34:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 13.768129026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X...............X.......] duration: 3.434586738s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:33:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.434614281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.584126974s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:33:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.584162302s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................X...............X......] duration: 3.485973301s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:32:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48600387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 14.547730417s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:32:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 14.547757006s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X................................X............] duration: 7.136871613s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:32:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.136897802s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X................................X...........] duration: 7.115843171s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:32:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.115873598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.647533142s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:31:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.647561587s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X...............X.....] duration: 3.369579125s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:31:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.36960346s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X...............X....] duration: 3.435232873s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:31:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.435256679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X................................X..........] duration: 6.668397063s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:31:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.66842639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X...............X...] duration: 3.389602835s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:31:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.389633101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.565738273s trace: https://jaeger.dev.storj.io/trace/6d728abedff5f55a FAIL (2025-12-08 11:31:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.565769815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.881616782s trace: https://jaeger.dev.storj.io/trace/6d728abedff5f55a FAIL (2025-12-08 11:30:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.881645437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.058871907s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:30:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.05890451s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)