Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 868744 reports (Page 11625 of 17375)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X........................................X....] duration: 3.33559147s trace: https://jaeger.dev.storj.io/trace/76a0e54d13f276f5 FAIL (2025-12-09 08:59:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.335621236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.692678618s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:59:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.692699679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X............X.........X................] duration: 5.805992526s trace: https://jaeger.dev.storj.io/trace/3a6fda9f00eec135 FAIL (2025-12-09 08:58:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.806017464s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X..........XX......................X..................] duration: 5.813862278s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:58:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.813891135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X..............X........................................X...] duration: 3.411944764s trace: https://jaeger.dev.storj.io/trace/76a0e54d13f276f5 FAIL (2025-12-09 08:58:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.411973499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.695903888s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:58:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.695950477s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X..........XX......................X.................] duration: 5.870837384s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:58:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.870852051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.902356253s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:57:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.902373635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X............X.........X...............] duration: 6.146279333s trace: https://jaeger.dev.storj.io/trace/3a6fda9f00eec135 FAIL (2025-12-09 08:57:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.146311644s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X..............X........................................X..] duration: 3.398951211s trace: https://jaeger.dev.storj.io/trace/76a0e54d13f276f5 FAIL (2025-12-09 08:57:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.398983552s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X..........XX......................X................] duration: 5.94903646s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:57:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.949064434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.604078243s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:57:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.604108781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X............X.........X..............] duration: 5.897544089s trace: https://jaeger.dev.storj.io/trace/3a6fda9f00eec135 FAIL (2025-12-09 08:57:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.897575127s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X..............X........................................X.] duration: 3.556450052s trace: https://jaeger.dev.storj.io/trace/76a0e54d13f276f5 FAIL (2025-12-09 08:57:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.556502593s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X..............X........................................X] duration: 5.001414155s trace: https://jaeger.dev.storj.io/trace/76a0e54d13f276f5 FAIL (2025-12-09 08:56:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001434063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X..........XX......................X...............] duration: 5.794012352s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:56:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.794080561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X............X.........X.............] duration: 5.625620352s trace: https://jaeger.dev.storj.io/trace/3a6fda9f00eec135 FAIL (2025-12-09 08:56:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.625647102s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.716611235s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:56:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.716636174s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X..........XX......................X..............] duration: 5.838808034s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:56:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.838835316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X..........XX......................X.............] duration: 5.808603912s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:55:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.808630734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X............X.........X............] duration: 5.864835674s trace: https://jaeger.dev.storj.io/trace/3a6fda9f00eec135 FAIL (2025-12-09 08:55:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.86488041s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X..........XX......................X............] duration: 5.677439981s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:55:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.677463095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X..............X........................................] duration: 3.486298529s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:55:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.486328236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.63616933s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:55:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.63620073s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X..............X.......................................] duration: 3.491299151s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:55:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491331111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.788431963s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:55:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.788467912s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X..........XX......................X...........] duration: 6.056594867s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:55:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.056624052s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X..............X......................................] duration: 3.445189689s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:54:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.445215928s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............X..........XX......................X..........] duration: 5.851556909s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:54:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.851918267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X............X.........X...........] duration: 5.899487719s trace: https://jaeger.dev.storj.io/trace/3a6fda9f00eec135 FAIL (2025-12-09 08:54:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.899517405s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X..............X.....................................] duration: 3.495536812s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:54:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49556231s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X..............X....................................] duration: 3.472950151s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:54:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.473081201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.683143786s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:54:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.683161269s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............X..........XX......................X.........] duration: 5.749519097s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:53:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.749549725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X..............X...................................] duration: 3.469242397s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:53:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469274689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X............X.........X..........] duration: 6.160245646s trace: https://jaeger.dev.storj.io/trace/3a6fda9f00eec135 FAIL (2025-12-09 08:53:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.160276585s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X...............X..........XX......................X........] duration: 5.759777499s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:53:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.75983056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.625203282s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:53:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.625230757s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X..............X..................................] duration: 3.528672888s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:52:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.528701864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X...............X..........XX......................X.......] duration: 5.706843128s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:52:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.706867324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.72446838s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:52:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.724503518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X............X.........X.........] duration: 5.912255183s trace: https://jaeger.dev.storj.io/trace/3a6fda9f00eec135 FAIL (2025-12-09 08:52:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.912366375s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X..............X.................................] duration: 3.53431308s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:52:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534364037s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X............X.........X........] duration: 5.73900212s trace: https://jaeger.dev.storj.io/trace/3a6fda9f00eec135 FAIL (2025-12-09 08:51:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.739029113s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..X...............X..........XX......................X......] duration: 5.819737424s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:51:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.819763404s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X..............X................................] duration: 3.493446289s trace: https://jaeger.dev.storj.io/trace/1794eb08423ddb90 FAIL (2025-12-09 08:51:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.493472258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.676168859s trace: https://jaeger.dev.storj.io/trace/45dd900b3925f1d FAIL (2025-12-09 08:51:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 08:21:54.805918589 (88h52m54.677001624s) [upload=77553]: Starting upload Dec 9 08:21:54.806299183 (88h52m54.677382218s) [upload=77553]: Done starting upload Dec 9 08:21:54.806426976 (88h52m54.677510021s) [upload=77553]: Uploading segments... Dec 9 08:21:54.806625885 (88h52m54.67770894s) [upload=77553]: Got next segment. Inline: false Dec 9 08:21:54.806676150 (88h52m54.677759195s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 08:21:54.806683154 (88h52m54.6777662s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 08:21:55.926705337 (88h52m55.797788382s) [seg_pos={0 0}, upload=77553]: Begin upload segment... Dec 9 08:21:55.926765221 (88h52m55.797848266s) [seg_pos={0 0}, upload=77553]: Done begin upload segment. Dec 9 08:21:55.926796871 (88h52m55.797879916s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.926805558 (88h52m55.797888603s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927331520 (88h52m55.798414565s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.927816042 (88h52m55.798899077s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.928268304 (88h52m55.799351349s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.930972443 (88h52m55.802055498s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931339652 (88h52m55.802422687s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.931949734 (88h52m55.803032779s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932310551 (88h52m55.803393596s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.932970678 (88h52m55.804053723s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.933363816 (88h52m55.804446861s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934703758 (88h52m55.805786803s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.934739967 (88h52m55.805823002s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935237454 (88h52m55.806320499s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935262051 (88h52m55.806345096s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:55.935787682 (88h52m55.806870727s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Uploading piece... Dec 9 08:21:57.629517740 (88h52m57.500600785s) [upload=77553]: Next returned nil segment Dec 9 08:21:57.629534602 (88h52m57.500617647s) [upload=77553]: Waiting for error group managing segments... Dec 9 08:21:57.915734340 (88h52m57.786817375s) [noise=true , address=145.79.227.27:4689 , piece_id=6ISRHMM6UA7CD6B7NK2M7B6MA5TEFGZNLHGG4ZRNDT5CRX2CPSUA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922651993 (88h52m57.793735038s) [noise=true , address=145.79.227.31:4798 , piece_id=IHAX2P3EUKZXZYONWIRYANI6UMCTOSKGSFERWPPTVOZWR3LP5DBA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.922726214 (88h52m57.793809259s) [noise=true , address=145.79.227.31:4797 , piece_id=GLMO3LVFIH34FCUW7D6KRGLZW2GFNCH6R2KHLT6UHPECCC7DCHJQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.924811665 (88h52m57.7958947s) [noise=true , address=145.79.227.27:4691 , piece_id=3KBH7RPL4HL7FVEC3BREZO3YJWU7N5NFNYWSY3HPAZKNFLR7NM3Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.927019118 (88h52m57.798102163s) [noise=true , address=145.79.227.22:4509 , piece_id=APGPITUOWW4ULVR7DEWP7YLY3O2JHTAR6U2LZZ2C4JNSMYMVTECA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.928941349 (88h52m57.800024384s) [noise=true , address=145.79.227.38:4977 , piece_id=UUY6DW7MZOVOF647HVJJOYQVCDAGSTLEH2L6PLZCNT7PHIGDBFOA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931583961 (88h52m57.802666997s) [noise=true , address=145.79.227.42:4122 , piece_id=MIHYX7CUKF7W5QQ7J37BQEKBPTXJIUHCDHSB3UYQSVSCMZ26JBPA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.931881878 (88h52m57.802964913s) [noise=true , address=145.79.227.26:4655 , piece_id=W5BG2RJ4LEG5LU4BOICLDGFCFVEXVD5KAJD76DBNY7TVEWYJLYAQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.939305485 (88h52m57.81038853s) [noise=true , address=145.79.227.24:4581 , piece_id=QODM5D6B3GAPCFGBIOUDJ7YWX4G7JD3BEPN5MB72XBTXGWNOGC6Q, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948447884 (88h52m57.819530919s) [noise=true , address=145.79.227.28:4726 , piece_id=SNZRMZ3IRU5ETZKFP4VRD3KCG2UZAXQANRZSX4PFU7IAPZQPSBPQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.948967323 (88h52m57.820050368s) [noise=true , address=145.79.227.21:4473 , piece_id=2ITYN2C4CDX3UFWB4EHPMDHBUVZTVBQ6YQVGKFSS2ZLU6J7DARZA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949024112 (88h52m57.820107147s) [noise=true , address=145.79.227.23:4546 , piece_id=5ODWCJD3QCBVPX5T7ASAHNEBLM7C7E3QMLIVQ342KD6OT5ET6P3A, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949415356 (88h52m57.820498401s) [noise=true , address=145.79.227.38:4979 , piece_id=NYV4WMC6VNZ6AS3LWQBYWVFCWKYUEQQC5MD5JZXMCXTTX3KQE2MA, seg_pos={0 0}, upload=77553]: Done uploading piece. err: <nil> Dec 9 08:21:57.949428942 (88h52m57.820511977s) [seg_pos={0 0}, upload=77553]: Segment reached optimal threshold of 13 pieces. Dec 9 08:21:57.949655153 (88h52m57.820738198s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949663529 (88h52m57.820746574s) [noise=true , address=145.79.227.20:4437 , piece_id=4Y7CUGVOXILD5PQCR6VEFHJNPEVQU7AFEDD5MYGILCXHUYQI5HKQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949704316 (88h52m57.820787371s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44002->77.90.31.13:4627: use of closed network connection Dec 9 08:21:57.949715397 (88h52m57.820798432s) [noise=true , address=77.90.31.13:4627 , piece_id=LTYMPGULYUQGIFM7G2RLHR772RDSRCR2VIDSH7HXREUHK2ZDERNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.949951066 (88h52m57.821034111s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 9 08:21:57.949967798 (88h52m57.821050843s) [noise=true , address=77.90.31.10:4519 , piece_id=CC547W7YCFSUV2THQHV6R2SJUVVA37345NWXWU2WZUJ6E56YWWNQ, seg_pos={0 0}, upload=77553]: Long tail cancellation due to optimal threshold Dec 9 08:21:57.950107084 (88h52m57.821190119s) [seg_pos={0 0}, upload=77553]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 08:21:57.950212204 (88h52m57.821295249s) [upload=77553]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 08:21:57.950217614 (88h52m57.821300659s) [upload=77553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 08:24:21.525107524 (88h55m21.39619057s) [upload=77553]: Done uploading segments... Dec 9 08:24:21.525408067 (88h55m21.396491122s) [upload=77553]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.676203044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X............X.........X.......] duration: 5.743656188s trace: https://jaeger.dev.storj.io/trace/3a6fda9f00eec135 FAIL (2025-12-09 08:50:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.743683149s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X...............X..........XX......................X.....] duration: 5.633132337s trace: https://jaeger.dev.storj.io/trace/5e8cc43df2e9fadd FAIL (2025-12-09 08:50:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.633162615s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X............X.........X......] duration: 5.709901632s trace: https://jaeger.dev.storj.io/trace/3a6fda9f00eec135 FAIL (2025-12-09 08:50:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.709938884s core uplink us1

1
...
11623
11624
11625
11626
11627
...
17375