Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X..............X........] duration: 3.570659337s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:34:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.570689704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 7.060243138s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:34:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.060272424s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........X.............X...............X..........XX.........] duration: 5.705851415s trace: https://jaeger.dev.storj.io/trace/1c2e4cb6840f75d FAIL (2025-12-09 08:33:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.705877845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................X..............X.......] duration: 3.446160565s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:33:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.446307275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.997510353s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:33:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.997573213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.932741751s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:33:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.932768821s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.314460971s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:33:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.314503051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........X.............X...............X..........XX........] duration: 6.271318771s trace: https://jaeger.dev.storj.io/trace/1c2e4cb6840f75d FAIL (2025-12-09 08:32:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.271345852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.199674645s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:32:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.199695467s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X.....................................X..............X......] duration: 3.647569909s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:32:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.647604546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.13507583s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:32:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.135104365s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........X.............X...............X..........XX.......] duration: 7.680776697s trace: https://jaeger.dev.storj.io/trace/1c2e4cb6840f75d FAIL (2025-12-09 08:32:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.680805873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.473344118s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:32:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.473366449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 11.885246995s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:32:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 11.885279706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.10136729s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:31:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.10139369s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X.....................................X..............X.....] duration: 3.973477963s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:31:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.973507461s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........X.............X...............X..........XX......] duration: 6.642806783s trace: https://jaeger.dev.storj.io/trace/1c2e4cb6840f75d FAIL (2025-12-09 08:31:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.64284755s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X.....................................X..............X....] duration: 4.401587634s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:31:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.40161705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.462632593s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:31:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.462666436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X.....................................X..............X...] duration: 3.549656555s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:30:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549700708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.991154802s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:30:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.991183778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.855392159s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:30:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.855423549s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............X.............X...............X..........XX.....] duration: 6.053169921s trace: https://jaeger.dev.storj.io/trace/1c2e4cb6840f75d FAIL (2025-12-09 08:30:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.053198497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.282054286s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:30:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.282085807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.2939867s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:30:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.294033821s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X.....................................X..............X..] duration: 3.485061842s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:30:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.485094215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............X.............X...............X..........XX....] duration: 5.821767457s trace: https://jaeger.dev.storj.io/trace/1c2e4cb6840f75d FAIL (2025-12-09 08:30:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.821799719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.671645527s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:29:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.671664635s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............X.............X...............X..........XX...] duration: 5.858341766s trace: https://jaeger.dev.storj.io/trace/1c2e4cb6840f75d FAIL (2025-12-09 08:29:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.858391651s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.83303106s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:29:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.833061708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.978753036s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:29:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.978782582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X............X.........] duration: 5.677520731s trace: https://jaeger.dev.storj.io/trace/332f068814d73b35 FAIL (2025-12-09 08:29:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.677547875s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X.....................................X..............X.] duration: 3.764837157s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:28:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.7648652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.813306086s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:28:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.813322889s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............X.............X...............X..........XX..] duration: 6.085310815s trace: https://jaeger.dev.storj.io/trace/1c2e4cb6840f75d FAIL (2025-12-09 08:28:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.085375198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X............X........] duration: 6.012165724s trace: https://jaeger.dev.storj.io/trace/332f068814d73b35 FAIL (2025-12-09 08:28:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.012191183s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.931937138s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:28:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.931979159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.221272292s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:28:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.221305875s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X.....................................X..............X] duration: 5.001525571s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:28:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001542302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.095974801s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:28:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.096006401s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X.....................................X..............] duration: 3.558066356s trace: https://jaeger.dev.storj.io/trace/22987434e2c92eb8 FAIL (2025-12-09 08:27:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.558093688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.455259859s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:27:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.45528656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................X.............X...............X..........XX.] duration: 5.836127796s trace: https://jaeger.dev.storj.io/trace/1c2e4cb6840f75d FAIL (2025-12-09 08:27:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.836188992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.642178422s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:27:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.642238296s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................X.............X...............X..........XX] duration: 1m0.001034316s trace: https://jaeger.dev.storj.io/trace/1c2e4cb6840f75d FAIL (2025-12-09 08:27:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001046631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X............X.......] duration: 5.750458323s trace: https://jaeger.dev.storj.io/trace/332f068814d73b35 FAIL (2025-12-09 08:27:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.750492057s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.645436629s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:27:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.645470473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.930702165s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:27:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.930722153s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X.....................................X.............] duration: 3.714167854s trace: https://jaeger.dev.storj.io/trace/22987434e2c92eb8 FAIL (2025-12-09 08:27:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.714197379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.467776577s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:26:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.467804552s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)