Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.706420227s trace: https://jaeger.dev.storj.io/trace/6d728abedff5f55a FAIL (2025-12-08 11:29:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.706487946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 3.020342141s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:29:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.020382018s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X................................X.......] duration: 7.115348045s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:29:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.115370499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................X...............X.] duration: 4.2890752s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:28:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.28911158s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 479.651617ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9b67b097dee132a FAIL (2025-12-08 11:28:58 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 479.678008ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X................................X......] duration: 39.367059509s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:28:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 39.367088193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 7.45354894s trace: https://jaeger.dev.storj.io/trace/6d728abedff5f55a FAIL (2025-12-08 11:28:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.453578636s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................X...............X] duration: 5.000660025s trace: https://jaeger.dev.storj.io/trace/724cd91e702c5e8e FAIL (2025-12-08 11:28:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000673932s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 466.604593ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9b67b097dee132a FAIL (2025-12-08 11:28:31 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.627277ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.820593194s trace: https://jaeger.dev.storj.io/trace/6d728abedff5f55a FAIL (2025-12-08 11:28:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.820619255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.258133536s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:27:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.258161578s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.481580266s trace: https://jaeger.dev.storj.io/trace/1fd386d4dc51b29a FAIL (2025-12-08 11:27:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481600543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X................................X.....] duration: 7.282797333s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:27:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.282828673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.598649814s trace: https://jaeger.dev.storj.io/trace/6d728abedff5f55a FAIL (2025-12-08 11:27:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.598747932s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 468.523438ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9b67b097dee132a FAIL (2025-12-08 11:27:26 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.584184ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.5369503s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:27:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.536978895s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 467.430647ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9b67b097dee132a FAIL (2025-12-08 11:26:59 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 467.457327ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X............................................X..............] duration: 3.404860357s trace: https://jaeger.dev.storj.io/trace/1fd386d4dc51b29a FAIL (2025-12-08 11:26:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.404926993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X................................X....] duration: 8.036246268s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:26:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.036270654s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 474.00629ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9b67b097dee132a FAIL (2025-12-08 11:26:33 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 474.032801ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.894535111s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:26:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.894567805s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.545006222s trace: https://jaeger.dev.storj.io/trace/6d728abedff5f55a FAIL (2025-12-08 11:26:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.545043452s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X............................................X.............] duration: 3.654675588s trace: https://jaeger.dev.storj.io/trace/1fd386d4dc51b29a FAIL (2025-12-08 11:26:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.654737164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.935166199s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:26:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.935199893s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X................................X...] duration: 8.171462418s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:25:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.171497345s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 474.697214ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9b67b097dee132a FAIL (2025-12-08 11:25:51 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 474.724346ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 3.083724796s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:25:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.08375358s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X............................................X............] duration: 3.473884764s trace: https://jaeger.dev.storj.io/trace/1fd386d4dc51b29a FAIL (2025-12-08 11:25:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.473946802s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.770876291s trace: https://jaeger.dev.storj.io/trace/6d728abedff5f55a FAIL (2025-12-08 11:25:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.770910676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X............................................X...........] duration: 3.449066063s trace: https://jaeger.dev.storj.io/trace/1fd386d4dc51b29a FAIL (2025-12-08 11:25:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.449083647s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 466.091979ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9b67b097dee132a FAIL (2025-12-08 11:24:52 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.122517ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X................................X..] duration: 7.812709889s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:24:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.812739075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.955807696s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:24:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.955836281s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X............................................X..........] duration: 3.36602113s trace: https://jaeger.dev.storj.io/trace/1fd386d4dc51b29a FAIL (2025-12-08 11:24:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.366048573s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.664113416s trace: https://jaeger.dev.storj.io/trace/6d728abedff5f55a FAIL (2025-12-08 11:24:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.664142513s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 474.744092ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9b67b097dee132a FAIL (2025-12-08 11:24:00 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 474.77463ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X............................................X.........] duration: 3.50142317s trace: https://jaeger.dev.storj.io/trace/1fd386d4dc51b29a FAIL (2025-12-08 11:23:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.501448148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X................................X.] duration: 6.818782194s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:23:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.818804858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X................................X] duration: 1m35.87125257s trace: https://jaeger.dev.storj.io/trace/2112bb1d86f60282 FAIL (2025-12-08 11:23:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914209602 (67h52m54.785292647s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.914793715 (67h52m54.78587676s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915307622 (67h52m54.786390668s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.915917094 (67h52m54.787000139s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.916606166 (67h52m54.787689211s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.918447993 (67h52m54.789531028s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.920151327 (67h52m54.791234372s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.921890178 (67h52m54.792973223s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.923995938 (67h52m54.795078973s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.926236043 (67h52m54.797319088s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928270907 (67h52m54.799353952s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.928760550 (67h52m54.799843595s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.930211603 (67h52m54.801294648s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.931821459 (67h52m54.802904494s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:54.932523997 (67h52m54.803607042s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Uploading piece... Dec 8 11:21:56.369615465 (67h52m56.24069851s) [upload=59513]: Next returned nil segment Dec 8 11:21:56.369661532 (67h52m56.240744567s) [upload=59513]: Waiting for error group managing segments... Dec 8 11:21:56.736135475 (67h52m56.60721852s) [noise=true , address=5.231.73.3:4001 , piece_id=DNLKLJYD4BW5THAF4OH6R6224G4YDX7VQBXGB6FJ4IFRUBF5RJDA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.913670270 (67h52m56.784753315s) [noise=true , address=77.90.31.18:4833 , piece_id=JFJRWZU5CFOZKCTHXRWJOO7N5T63MCQO5OURBY35BPGUYKQF2YQA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.916167245 (67h52m56.78725029s) [noise=true , address=77.90.31.11:4558 , piece_id=NLAKLQXVR5FBEOTMYESHVGZ7LMLMKQGYUY4IGOVAS6UTIM2MFW3A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.920203792 (67h52m56.791286837s) [noise=true , address=77.90.31.21:4908 , piece_id=ATPU55BHANNT3LK4SN7VDKCYJ4BTFDSSZZF4SORXFBMIVIZFHAXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.940489100 (67h52m56.811572135s) [noise=true , address=77.90.31.14:4687 , piece_id=7RVEFCXHRUPYD5FEIWUW54Y24DGYDGDGCD4RWDIPLRZBMLHSTSFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.959648452 (67h52m56.830731487s) [noise=true , address=77.90.31.15:4702 , piece_id=NLMTHTY3XQ6GOHHZLAMB7LPVS2WMXC3PFJD6ZXUHDX23E6NSKIFA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.965342996 (67h52m56.836426031s) [noise=true , address=77.90.31.12:4623 , piece_id=PKVIAYDNTY5DKWUB5332KXVPSE5WIBKXPLBS4H4GCWALTAUPPTLA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.984330944 (67h52m56.855413989s) [noise=true , address=45.91.116.10:4089 , piece_id=ML2LPZZP5E7LZ4IMAN65ZJ25ZDQBTRMWCZ5A22XTRSHMR6NPS6AQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.986436073 (67h52m56.857519118s) [noise=true , address=45.91.116.5:4304 , piece_id=LW3EMHCFVK4HU2XE3QJ5UCE6FGJPEDBQBXBGWCEL2SRPUDYRIPXA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:56.989448639 (67h52m56.860531684s) [noise=true , address=45.91.116.4:4969 , piece_id=K4MAALGOPZYKW6J27OG6H6CQQ6CT7MCBUL6NHMRLGHCBIJJVQPVQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.028239525 (67h52m56.89932256s) [noise=true , address=45.91.116.8:4046 , piece_id=C2B3C3DH5YAPSUTUKSIDGC4CVAPEPRCEG3LXVDATW2PJXXVSS5CA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.070969141 (67h52m56.942052186s) [noise=true , address=77.90.31.19:4873 , piece_id=RM3FUZQS47HWEAPGZEEUYA7VCA2XS5JNCQTGVFESFVIWI3KFDBSQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.124406888 (67h52m56.995489933s) [noise=true , address=45.91.116.6:4003 , piece_id=BY5MMPH7OYZ6R3FT2S7TJFX52MKZDHE5JOFB2YLL3I5ELSY56UJQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427950218 (67h52m57.299033254s) [noise=true , address=145.79.227.15:4908 , piece_id=YZNSWWNDHHYPURO4UOSF5URRGGNUSLYS5SGBBK3UD5YKW2SU4MOQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: <nil> Dec 8 11:21:57.427969997 (67h52m57.299053042s) [seg_pos={0 0}, upload=59513]: Segment reached optimal threshold of 14 pieces. Dec 8 11:21:57.428127125 (67h52m57.29921016s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428133828 (67h52m57.299216863s) [noise=true , address=145.79.227.25:4649 , piece_id=KXI6PC2LWSCX723AU37H2SPIJJKHJNMYA2DFWRECO3BGQS6KQD7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428307929 (67h52m57.299390974s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq): protocol: rpc: dial tcp 77.90.31.17:4788: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 11:21:57.428313371 (67h52m57.299396416s) [noise=true , address=77.90.31.17:4788 , piece_id=K45FLLTEVB4E6XEMLUJN5R2VQVNVAT7JQENTTKCUMRHMJ75CLBTA, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428386349 (67h52m57.299469394s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57948->64.49.13.169:4707: use of closed network connection Dec 8 11:21:57.428394785 (67h52m57.29947782s) [noise=true , address=64.49.13.169:4707 , piece_id=SF23JHIDC6UQONQOI6VY7F43P2OMYXNXKUAGQZUDBAIZO44PZL7A, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428461502 (67h52m57.299544537s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 8 11:21:57.428468005 (67h52m57.29955104s) [noise=true , address=145.79.227.30:4774 , piece_id=HW6LQ42XXQQCHA5JNN4H3KI5WSNGAHKLPPCBU4JT62ZKR76C2CEQ, seg_pos={0 0}, upload=59513]: Long tail cancellation due to optimal threshold Dec 8 11:21:57.428489907 (67h52m57.299572942s) [seg_pos={0 0}, upload=59513]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:21:57.428651494 (67h52m57.299734529s) [upload=59513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:21:57.428657676 (67h52m57.299740721s) [upload=59513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:23:25.170470186 (67h54m25.041553222s) [upload=59513]: Done uploading segments... Dec 8 11:23:25.170555429 (67h54m25.041638464s) [upload=59513]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m35.871280542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.517608279s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:23:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.517634258s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.669267457s trace: https://jaeger.dev.storj.io/trace/6d728abedff5f55a FAIL (2025-12-08 11:23:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.669294238s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 468.15766ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9b67b097dee132a FAIL (2025-12-08 11:23:02 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.185294ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 3.045826589s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:22:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.045850424s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X............................................X........] duration: 3.466900607s trace: https://jaeger.dev.storj.io/trace/1fd386d4dc51b29a FAIL (2025-12-08 11:22:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466931766s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.960029865s trace: https://jaeger.dev.storj.io/trace/6d728abedff5f55a FAIL (2025-12-08 11:22:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.960066024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.450871041s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:22:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.450891119s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 466.586979ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9b67b097dee132a FAIL (2025-12-08 11:21:51 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.611937ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X............................................X.......] duration: 3.41854908s trace: https://jaeger.dev.storj.io/trace/1fd386d4dc51b29a FAIL (2025-12-08 11:21:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.41857489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.602792087s trace: https://jaeger.dev.storj.io/trace/39dd084a0bc7a92d FAIL (2025-12-08 11:21:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.602819268s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 7.287448628s trace: https://jaeger.dev.storj.io/trace/5238c014c248ae80 FAIL (2025-12-08 11:21:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ANQ, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.437987094 (67h31m6.309070139s) [noise=true , address=77.90.31.18:4839 , piece_id=KWTYKFCQPWGGHV36J43456YOM2ATUR72GBQZOHJFKIL7MVXFG5MA, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.438568261 (67h31m6.309651306s) [noise=true , address=77.90.31.21:4908 , piece_id=6QMOXHJT6WTDYFUSY2OPEAMHPFSPNSCQW3JPRDWRNK2UW5OTJM3A, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.439013669 (67h31m6.310096724s) [noise=true , address=5.231.73.2:4001 , piece_id=ASM2FGIEC52M7HFT5YOC3L3IJDBTYIYDCA44YAW4UKRQMZQVUM4Q, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.439562925 (67h31m6.31064597s) [noise=true , address=77.90.31.13:4636 , piece_id=YM6ZUEIVLAZ442KTMMK77SK4WARJI3BEFSW5HIBP77GS4VJMHUCA, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.440027359 (67h31m6.311110404s) [noise=true , address=77.90.31.19:4847 , piece_id=JFR66BZOF3BZZER4UDHGSR5J5GXRDZECI4AWGI4OFIVTI6PSQ4GA, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.440738102 (67h31m6.311821157s) [noise=true , address=77.90.31.11:4573 , piece_id=JW6NMT2PIHU5KPMRWF77LOBLHM4ONKI5MPXNU4PAIEXNHPOVHYEQ, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.441218718 (67h31m6.312301773s) [noise=true , address=64.49.13.169:4706 , piece_id=4G2EOZA6DSANCRUTHYSFUMH7U2JPQKSCA4VPWP7PFYDZCDPPXHJA, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.441700004 (67h31m6.312783049s) [noise=true , address=45.91.116.8:4077 , piece_id=6CLBZFKEUYBHLNVHRPUXKLNE2YTOGN7DCVX3W5QNCO3REQYUKGWA, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.442210376 (67h31m6.313293432s) [noise=true , address=77.90.31.10:4542 , piece_id=PSEI2AI56LA6CHMLLY4772IXQC2IXU3RXJYXTA3HC4QNBFLDRGVQ, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.442763450 (67h31m6.313846495s) [noise=true , address=77.90.31.15:4730 , piece_id=B6KZ3AK5FF2LMLBVVGY6TFNDRDMOC3BMDL3Q7CFHMQVADX3QR4AQ, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.443314379 (67h31m6.314397424s) [noise=true , address=45.91.116.6:4005 , piece_id=WTSSEF6DKWEEGY57D7DZJNVKU4SWJFCDWCZ5DV2SLXRA4KB33QOQ, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.443869034 (67h31m6.314952109s) [noise=true , address=77.90.31.16:4755 , piece_id=OZ52IAD3LODF2KGEFQJC4H6XOQNZUBYZFR6BPDRUSUR3GGUB6MLQ, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.444385868 (67h31m6.315468914s) [noise=true , address=45.91.116.12:4251 , piece_id=7Y5AZRVKL2EX3GT4VFWOBQCYDPHZCYINAG47QHY7KLZOIQL2X56A, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.444904436 (67h31m6.315987481s) [noise=true , address=145.79.227.31:4824 , piece_id=VM522THFWVWCQCSVQFYXFHIDZPBWVUAGYQTP3XHXVP6OS46ZMGQQ, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.447912074 (67h31m6.318995129s) [noise=true , address=77.90.31.12:4603 , piece_id=AWKIQNN4NBO32FZWYOIURUDRTTGQ4RUC6VTO6BN66P6V6MVZIZPA, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:06.448682511 (67h31m6.319765556s) [noise=true , address=77.90.31.17:4782 , piece_id=EVZ5YBWYB3GMG3RKXO5XQG3D26DVDCM3C5O3TFDKOP2KNSVRTDHA, seg_pos={0 0}, upload=59196]: Uploading piece... Dec 8 11:00:07.935303466 (67h31m7.806386511s) [upload=59196]: Next returned nil segment Dec 8 11:00:07.935324596 (67h31m7.806407641s) [upload=59196]: Waiting for error group managing segments... Dec 8 11:00:08.409597368 (67h31m8.280680443s) [noise=true , address=5.231.73.2:4001 , piece_id=ASM2FGIEC52M7HFT5YOC3L3IJDBTYIYDCA44YAW4UKRQMZQVUM4Q, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.490922888 (67h31m8.362005943s) [noise=true , address=77.90.31.12:4603 , piece_id=AWKIQNN4NBO32FZWYOIURUDRTTGQ4RUC6VTO6BN66P6V6MVZIZPA, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.491791852 (67h31m8.362874898s) [noise=true , address=77.90.31.10:4542 , piece_id=PSEI2AI56LA6CHMLLY4772IXQC2IXU3RXJYXTA3HC4QNBFLDRGVQ, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.495495926 (67h31m8.366578971s) [noise=true , address=77.90.31.15:4730 , piece_id=B6KZ3AK5FF2LMLBVVGY6TFNDRDMOC3BMDL3Q7CFHMQVADX3QR4AQ, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.506318194 (67h31m8.377401239s) [noise=true , address=45.91.116.9:4266 , piece_id=7EXTSTS7GJ2E5PFPUT5RXEKPTDP7BZIG25P26KV5HWQV3AB26ANQ, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.518184790 (67h31m8.389267825s) [noise=true , address=77.90.31.19:4847 , piece_id=JFR66BZOF3BZZER4UDHGSR5J5GXRDZECI4AWGI4OFIVTI6PSQ4GA, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.518816343 (67h31m8.389899389s) [noise=true , address=77.90.31.21:4908 , piece_id=6QMOXHJT6WTDYFUSY2OPEAMHPFSPNSCQW3JPRDWRNK2UW5OTJM3A, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.551524195 (67h31m8.42260724s) [noise=true , address=77.90.31.18:4839 , piece_id=KWTYKFCQPWGGHV36J43456YOM2ATUR72GBQZOHJFKIL7MVXFG5MA, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.563222340 (67h31m8.434305386s) [noise=true , address=45.91.116.12:4251 , piece_id=7Y5AZRVKL2EX3GT4VFWOBQCYDPHZCYINAG47QHY7KLZOIQL2X56A, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.567338489 (67h31m8.438421534s) [noise=true , address=77.90.31.13:4636 , piece_id=YM6ZUEIVLAZ442KTMMK77SK4WARJI3BEFSW5HIBP77GS4VJMHUCA, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.589327861 (67h31m8.460410906s) [noise=true , address=77.90.31.11:4573 , piece_id=JW6NMT2PIHU5KPMRWF77LOBLHM4ONKI5MPXNU4PAIEXNHPOVHYEQ, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.590361249 (67h31m8.461444294s) [noise=true , address=77.90.31.16:4755 , piece_id=OZ52IAD3LODF2KGEFQJC4H6XOQNZUBYZFR6BPDRUSUR3GGUB6MLQ, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.612737908 (67h31m8.483820953s) [noise=true , address=45.91.116.6:4005 , piece_id=WTSSEF6DKWEEGY57D7DZJNVKU4SWJFCDWCZ5DV2SLXRA4KB33QOQ, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.637773241 (67h31m8.508856286s) [noise=true , address=45.91.116.8:4077 , piece_id=6CLBZFKEUYBHLNVHRPUXKLNE2YTOGN7DCVX3W5QNCO3REQYUKGWA, seg_pos={0 0}, upload=59196]: Done uploading piece. err: <nil> Dec 8 11:00:08.637864515 (67h31m8.50894756s) [seg_pos={0 0}, upload=59196]: Segment reached optimal threshold of 14 pieces. Dec 8 11:00:08.638128958 (67h31m8.509212003s) [noise=true , address=77.90.31.17:4782 , piece_id=EVZ5YBWYB3GMG3RKXO5XQG3D26DVDCM3C5O3TFDKOP2KNSVRTDHA, seg_pos={0 0}, upload=59196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk): protocol: expected piece hash; context canceled; EOF Dec 8 11:00:08.638141432 (67h31m8.509224477s) [noise=true , address=77.90.31.17:4782 , piece_id=EVZ5YBWYB3GMG3RKXO5XQG3D26DVDCM3C5O3TFDKOP2KNSVRTDHA, seg_pos={0 0}, upload=59196]: Long tail cancellation due to optimal threshold Dec 8 11:00:08.638315724 (67h31m8.509398769s) [noise=true , address=145.79.227.32:4863 , piece_id=ZECUWHATYCNNMNKKXYP6LMFWO2OSP3V7KRBFAAR5NDPZRLO2DN6Q, seg_pos={0 0}, upload=59196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hoK1e9x7ms16dWqXpNpfHA97U74iX9DkT7n73YUTcePgbMatZ): protocol: expected piece hash; context canceled; EOF Dec 8 11:00:08.638329390 (67h31m8.509412485s) [noise=true , address=145.79.227.32:4863 , piece_id=ZECUWHATYCNNMNKKXYP6LMFWO2OSP3V7KRBFAAR5NDPZRLO2DN6Q, seg_pos={0 0}, upload=59196]: Long tail cancellation due to optimal threshold Dec 8 11:00:08.638577612 (67h31m8.509660657s) [noise=true , address=64.49.13.169:4706 , piece_id=4G2EOZA6DSANCRUTHYSFUMH7U2JPQKSCA4VPWP7PFYDZCDPPXHJA, seg_pos={0 0}, upload=59196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:41372->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:41372->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41372->64.49.13.169:4706: use of closed network connection Dec 8 11:00:08.638598763 (67h31m8.509681808s) [noise=true , address=64.49.13.169:4706 , piece_id=4G2EOZA6DSANCRUTHYSFUMH7U2JPQKSCA4VPWP7PFYDZCDPPXHJA, seg_pos={0 0}, upload=59196]: Long tail cancellation due to optimal threshold Dec 8 11:00:08.638625183 (67h31m8.509708229s) [noise=true , address=145.79.227.31:4824 , piece_id=VM522THFWVWCQCSVQFYXFHIDZPBWVUAGYQTP3XHXVP6OS46ZMGQQ, seg_pos={0 0}, upload=59196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QTjnomQYFETHbaJBWxWEPo5yweArk68bKPeooZXpRZrpw1ACq): protocol: expected piece hash; context canceled; EOF Dec 8 11:00:08.638638518 (67h31m8.509721564s) [noise=true , address=145.79.227.31:4824 , piece_id=VM522THFWVWCQCSVQFYXFHIDZPBWVUAGYQTP3XHXVP6OS46ZMGQQ, seg_pos={0 0}, upload=59196]: Long tail cancellation due to optimal threshold Dec 8 11:00:08.638666302 (67h31m8.509749347s) [seg_pos={0 0}, upload=59196]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 11:00:08.638772143 (67h31m8.509855188s) [upload=59196]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 11:00:08.638779918 (67h31m8.509862963s) [upload=59196]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 11:01:27.468435151 (67h32m27.339518186s) [upload=59196]: Done uploading segments... Dec 8 11:01:27.468600105 (67h32m27.33968315s) [upload=59196]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.287517348s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)