Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 9.56780326s trace: https://jaeger.dev.storj.io/trace/55a82266a480eaa9 FAIL (2025-12-15 11:53:10 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881615EFFF0BABD, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 9.567828059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...................X.....] duration: 10.036630357s trace: https://jaeger.dev.storj.io/trace/328ec34d2eb5b93c FAIL (2025-12-15 11:52:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 11:48:04.267806751 (236h19m4.138889797s) [upload=207194]: Starting upload Dec 15 11:48:04.267881564 (236h19m4.138964609s) [upload=207194]: Done starting upload Dec 15 11:48:04.267964322 (236h19m4.139047367s) [upload=207194]: Uploading segments... Dec 15 11:48:04.268116571 (236h19m4.139199617s) [upload=207194]: Got next segment. Inline: false Dec 15 11:48:04.268155566 (236h19m4.139238611s) [upload=207194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:48:04.268161216 (236h19m4.139244261s) [upload=207194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:48:04.645157786 (236h19m4.516240831s) [seg_pos={0 0}, upload=207194]: Begin upload segment... Dec 15 11:48:04.645232939 (236h19m4.516315984s) [seg_pos={0 0}, upload=207194]: Done begin upload segment. Dec 15 11:48:04.645355942 (236h19m4.516438988s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.645373907 (236h19m4.516456942s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646387267 (236h19m4.517470312s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646492347 (236h19m4.517575392s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647115493 (236h19m4.518198549s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647917941 (236h19m4.519000986s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.648686515 (236h19m4.51976956s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.649098740 (236h19m4.520181795s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650330255 (236h19m4.5214133s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650556706 (236h19m4.521639751s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650759122 (236h19m4.521842167s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651119207 (236h19m4.522202252s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651345759 (236h19m4.522428804s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651571018 (236h19m4.522654053s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651948907 (236h19m4.523031952s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.652055821 (236h19m4.523138866s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:05.780921391 (236h19m5.652004436s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.074553880 (236h19m5.945636935s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.773692793 (236h19m6.644775838s) [upload=207194]: Next returned nil segment Dec 15 11:48:06.773722018 (236h19m6.644805053s) [upload=207194]: Waiting for error group managing segments... Dec 15 11:48:07.197011248 (236h19m7.068094293s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:07.686745571 (236h19m7.557828616s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:10.935626099 (236h19m10.806709144s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.390197256 (236h19m11.261280301s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.484228721 (236h19m11.355311766s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.850925194 (236h19m11.722008239s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:15.325714041 (236h19m15.196797086s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:16.269216292 (236h19m16.140299337s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:17.377248139 (236h19m17.248331184s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:20.448167344 (236h19m20.319250379s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:26.357864502 (236h19m26.228947547s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:27.921052250 (236h19m27.792135285s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:33.166255103 (236h19m33.037338138s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166426560 (236h19m33.037510236s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166523425 (236h19m33.03760646s) [seg_pos={0 0}, upload=207194]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 11:48:33.542478980 (236h19m33.413562025s) [upload=207194]: Done uploading segments... Dec 15 11:48:33.542516501 (236h19m33.413599536s) [upload=207194]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.036660654s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 8.311580313s trace: https://jaeger.dev.storj.io/trace/6ab8ad2c2b59d9e5 FAIL (2025-12-15 11:52:40 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161580CA0AB6F, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 8.311607216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [......X.............X........X.X.X....X...........X..XXXXXXX] duration: 5.00110443s trace: https://jaeger.dev.storj.io/trace/76b84bf75aaea32e FAIL (2025-12-15 11:52:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001127765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...................X....] duration: 14.351313417s trace: https://jaeger.dev.storj.io/trace/328ec34d2eb5b93c FAIL (2025-12-15 11:52:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 11:48:04.267806751 (236h19m4.138889797s) [upload=207194]: Starting upload Dec 15 11:48:04.267881564 (236h19m4.138964609s) [upload=207194]: Done starting upload Dec 15 11:48:04.267964322 (236h19m4.139047367s) [upload=207194]: Uploading segments... Dec 15 11:48:04.268116571 (236h19m4.139199617s) [upload=207194]: Got next segment. Inline: false Dec 15 11:48:04.268155566 (236h19m4.139238611s) [upload=207194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:48:04.268161216 (236h19m4.139244261s) [upload=207194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:48:04.645157786 (236h19m4.516240831s) [seg_pos={0 0}, upload=207194]: Begin upload segment... Dec 15 11:48:04.645232939 (236h19m4.516315984s) [seg_pos={0 0}, upload=207194]: Done begin upload segment. Dec 15 11:48:04.645355942 (236h19m4.516438988s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.645373907 (236h19m4.516456942s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646387267 (236h19m4.517470312s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646492347 (236h19m4.517575392s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647115493 (236h19m4.518198549s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647917941 (236h19m4.519000986s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.648686515 (236h19m4.51976956s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.649098740 (236h19m4.520181795s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650330255 (236h19m4.5214133s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650556706 (236h19m4.521639751s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650759122 (236h19m4.521842167s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651119207 (236h19m4.522202252s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651345759 (236h19m4.522428804s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651571018 (236h19m4.522654053s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651948907 (236h19m4.523031952s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.652055821 (236h19m4.523138866s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:05.780921391 (236h19m5.652004436s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.074553880 (236h19m5.945636935s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.773692793 (236h19m6.644775838s) [upload=207194]: Next returned nil segment Dec 15 11:48:06.773722018 (236h19m6.644805053s) [upload=207194]: Waiting for error group managing segments... Dec 15 11:48:07.197011248 (236h19m7.068094293s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:07.686745571 (236h19m7.557828616s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:10.935626099 (236h19m10.806709144s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.390197256 (236h19m11.261280301s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.484228721 (236h19m11.355311766s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.850925194 (236h19m11.722008239s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:15.325714041 (236h19m15.196797086s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:16.269216292 (236h19m16.140299337s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:17.377248139 (236h19m17.248331184s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:20.448167344 (236h19m20.319250379s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:26.357864502 (236h19m26.228947547s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:27.921052250 (236h19m27.792135285s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:33.166255103 (236h19m33.037338138s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166426560 (236h19m33.037510236s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166523425 (236h19m33.03760646s) [seg_pos={0 0}, upload=207194]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 11:48:33.542478980 (236h19m33.413562025s) [upload=207194]: Done uploading segments... Dec 15 11:48:33.542516501 (236h19m33.413599536s) [upload=207194]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.351397398s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 10.116754498s trace: https://jaeger.dev.storj.io/trace/7f49278ca8522730 FAIL (2025-12-15 11:52:27 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 18816154F9972B60, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 10.116785198s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 10.581097447s trace: https://jaeger.dev.storj.io/trace/5d394a17d3090c7a FAIL (2025-12-15 11:51:56 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881614DDD466500, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 10.581125251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.......X.............X........X.X.X....X...........X..XXXXXX] duration: 5.000870294s trace: https://jaeger.dev.storj.io/trace/9a4132bfeca28c FAIL (2025-12-15 11:51:26 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 5.000893408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...................X...] duration: 10.804643215s trace: https://jaeger.dev.storj.io/trace/328ec34d2eb5b93c FAIL (2025-12-15 11:51:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 11:48:04.267806751 (236h19m4.138889797s) [upload=207194]: Starting upload Dec 15 11:48:04.267881564 (236h19m4.138964609s) [upload=207194]: Done starting upload Dec 15 11:48:04.267964322 (236h19m4.139047367s) [upload=207194]: Uploading segments... Dec 15 11:48:04.268116571 (236h19m4.139199617s) [upload=207194]: Got next segment. Inline: false Dec 15 11:48:04.268155566 (236h19m4.139238611s) [upload=207194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:48:04.268161216 (236h19m4.139244261s) [upload=207194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:48:04.645157786 (236h19m4.516240831s) [seg_pos={0 0}, upload=207194]: Begin upload segment... Dec 15 11:48:04.645232939 (236h19m4.516315984s) [seg_pos={0 0}, upload=207194]: Done begin upload segment. Dec 15 11:48:04.645355942 (236h19m4.516438988s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.645373907 (236h19m4.516456942s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646387267 (236h19m4.517470312s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646492347 (236h19m4.517575392s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647115493 (236h19m4.518198549s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647917941 (236h19m4.519000986s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.648686515 (236h19m4.51976956s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.649098740 (236h19m4.520181795s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650330255 (236h19m4.5214133s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650556706 (236h19m4.521639751s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650759122 (236h19m4.521842167s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651119207 (236h19m4.522202252s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651345759 (236h19m4.522428804s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651571018 (236h19m4.522654053s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651948907 (236h19m4.523031952s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.652055821 (236h19m4.523138866s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:05.780921391 (236h19m5.652004436s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.074553880 (236h19m5.945636935s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.773692793 (236h19m6.644775838s) [upload=207194]: Next returned nil segment Dec 15 11:48:06.773722018 (236h19m6.644805053s) [upload=207194]: Waiting for error group managing segments... Dec 15 11:48:07.197011248 (236h19m7.068094293s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:07.686745571 (236h19m7.557828616s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:10.935626099 (236h19m10.806709144s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.390197256 (236h19m11.261280301s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.484228721 (236h19m11.355311766s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.850925194 (236h19m11.722008239s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:15.325714041 (236h19m15.196797086s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:16.269216292 (236h19m16.140299337s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:17.377248139 (236h19m17.248331184s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:20.448167344 (236h19m20.319250379s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:26.357864502 (236h19m26.228947547s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:27.921052250 (236h19m27.792135285s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:33.166255103 (236h19m33.037338138s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166426560 (236h19m33.037510236s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166523425 (236h19m33.03760646s) [seg_pos={0 0}, upload=207194]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 11:48:33.542478980 (236h19m33.413562025s) [upload=207194]: Done uploading segments... Dec 15 11:48:33.542516501 (236h19m33.413599536s) [upload=207194]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.804659896s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 9.467558205s trace: https://jaeger.dev.storj.io/trace/7815e4919930310 FAIL (2025-12-15 11:50:52 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881613EEE32A626, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 9.467581518s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 11.999605989s trace: https://jaeger.dev.storj.io/trace/4203e2f34bd3cef FAIL (2025-12-15 11:50:31 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881613A07C94C7F, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 11.999630186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [........X.............X........X.X.X....X...........X..XXXXX] duration: 5.000408795s trace: https://jaeger.dev.storj.io/trace/38e4c46cfad10fbc FAIL (2025-12-15 11:50:30 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 5.000434143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...................X..] duration: 9.66843345s trace: https://jaeger.dev.storj.io/trace/328ec34d2eb5b93c FAIL (2025-12-15 11:50:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 11:48:04.267806751 (236h19m4.138889797s) [upload=207194]: Starting upload Dec 15 11:48:04.267881564 (236h19m4.138964609s) [upload=207194]: Done starting upload Dec 15 11:48:04.267964322 (236h19m4.139047367s) [upload=207194]: Uploading segments... Dec 15 11:48:04.268116571 (236h19m4.139199617s) [upload=207194]: Got next segment. Inline: false Dec 15 11:48:04.268155566 (236h19m4.139238611s) [upload=207194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:48:04.268161216 (236h19m4.139244261s) [upload=207194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:48:04.645157786 (236h19m4.516240831s) [seg_pos={0 0}, upload=207194]: Begin upload segment... Dec 15 11:48:04.645232939 (236h19m4.516315984s) [seg_pos={0 0}, upload=207194]: Done begin upload segment. Dec 15 11:48:04.645355942 (236h19m4.516438988s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.645373907 (236h19m4.516456942s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646387267 (236h19m4.517470312s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646492347 (236h19m4.517575392s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647115493 (236h19m4.518198549s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647917941 (236h19m4.519000986s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.648686515 (236h19m4.51976956s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.649098740 (236h19m4.520181795s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650330255 (236h19m4.5214133s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650556706 (236h19m4.521639751s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650759122 (236h19m4.521842167s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651119207 (236h19m4.522202252s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651345759 (236h19m4.522428804s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651571018 (236h19m4.522654053s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651948907 (236h19m4.523031952s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.652055821 (236h19m4.523138866s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:05.780921391 (236h19m5.652004436s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.074553880 (236h19m5.945636935s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.773692793 (236h19m6.644775838s) [upload=207194]: Next returned nil segment Dec 15 11:48:06.773722018 (236h19m6.644805053s) [upload=207194]: Waiting for error group managing segments... Dec 15 11:48:07.197011248 (236h19m7.068094293s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:07.686745571 (236h19m7.557828616s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:10.935626099 (236h19m10.806709144s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.390197256 (236h19m11.261280301s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.484228721 (236h19m11.355311766s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.850925194 (236h19m11.722008239s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:15.325714041 (236h19m15.196797086s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:16.269216292 (236h19m16.140299337s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:17.377248139 (236h19m17.248331184s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:20.448167344 (236h19m20.319250379s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:26.357864502 (236h19m26.228947547s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:27.921052250 (236h19m27.792135285s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:33.166255103 (236h19m33.037338138s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166426560 (236h19m33.037510236s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166523425 (236h19m33.03760646s) [seg_pos={0 0}, upload=207194]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 11:48:33.542478980 (236h19m33.413562025s) [upload=207194]: Done uploading segments... Dec 15 11:48:33.542516501 (236h19m33.413599536s) [upload=207194]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.668464821s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.........X.............X........X.X.X....X...........X..XXXX] duration: 5.000857318s trace: https://jaeger.dev.storj.io/trace/25aaedebc51f6a99 FAIL (2025-12-15 11:50:03 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 5.000998396s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..........X.............X........X.X.X....X...........X..XXX] duration: 5.000326274s trace: https://jaeger.dev.storj.io/trace/74d12ed78405d5d6 FAIL (2025-12-15 11:49:39 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 5.000365358s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 10.988558454s trace: https://jaeger.dev.storj.io/trace/8dd8c619085d803 FAIL (2025-12-15 11:49:26 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881612B0728EE84, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 10.988584694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...........X.............X........X.X.X....X...........X..XX] duration: 5.000319865s trace: https://jaeger.dev.storj.io/trace/154ea7923d2ea3c FAIL (2025-12-15 11:49:13 UTC) orbiter warning: operation error S3: PutObject, request canceled, context deadline exceeded Duration: 5.000341285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 7.347653335s trace: https://jaeger.dev.storj.io/trace/24c206e8447d7bd3 FAIL (2025-12-15 11:49:06 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161264C28E63F, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 7.347699924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...................X.] duration: 13.793611219s trace: https://jaeger.dev.storj.io/trace/328ec34d2eb5b93c FAIL (2025-12-15 11:49:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 11:48:04.267806751 (236h19m4.138889797s) [upload=207194]: Starting upload Dec 15 11:48:04.267881564 (236h19m4.138964609s) [upload=207194]: Done starting upload Dec 15 11:48:04.267964322 (236h19m4.139047367s) [upload=207194]: Uploading segments... Dec 15 11:48:04.268116571 (236h19m4.139199617s) [upload=207194]: Got next segment. Inline: false Dec 15 11:48:04.268155566 (236h19m4.139238611s) [upload=207194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:48:04.268161216 (236h19m4.139244261s) [upload=207194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:48:04.645157786 (236h19m4.516240831s) [seg_pos={0 0}, upload=207194]: Begin upload segment... Dec 15 11:48:04.645232939 (236h19m4.516315984s) [seg_pos={0 0}, upload=207194]: Done begin upload segment. Dec 15 11:48:04.645355942 (236h19m4.516438988s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.645373907 (236h19m4.516456942s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646387267 (236h19m4.517470312s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646492347 (236h19m4.517575392s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647115493 (236h19m4.518198549s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647917941 (236h19m4.519000986s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.648686515 (236h19m4.51976956s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.649098740 (236h19m4.520181795s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650330255 (236h19m4.5214133s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650556706 (236h19m4.521639751s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650759122 (236h19m4.521842167s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651119207 (236h19m4.522202252s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651345759 (236h19m4.522428804s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651571018 (236h19m4.522654053s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651948907 (236h19m4.523031952s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.652055821 (236h19m4.523138866s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:05.780921391 (236h19m5.652004436s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.074553880 (236h19m5.945636935s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.773692793 (236h19m6.644775838s) [upload=207194]: Next returned nil segment Dec 15 11:48:06.773722018 (236h19m6.644805053s) [upload=207194]: Waiting for error group managing segments... Dec 15 11:48:07.197011248 (236h19m7.068094293s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:07.686745571 (236h19m7.557828616s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:10.935626099 (236h19m10.806709144s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.390197256 (236h19m11.261280301s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.484228721 (236h19m11.355311766s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.850925194 (236h19m11.722008239s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:15.325714041 (236h19m15.196797086s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:16.269216292 (236h19m16.140299337s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:17.377248139 (236h19m17.248331184s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:20.448167344 (236h19m20.319250379s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:26.357864502 (236h19m26.228947547s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:27.921052250 (236h19m27.792135285s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:33.166255103 (236h19m33.037338138s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166426560 (236h19m33.037510236s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166523425 (236h19m33.03760646s) [seg_pos={0 0}, upload=207194]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 11:48:33.542478980 (236h19m33.413562025s) [upload=207194]: Done uploading segments... Dec 15 11:48:33.542516501 (236h19m33.413599536s) [upload=207194]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.793630095s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [............X.............X........X.X.X....X...........X..X] duration: 5.00108952s trace: https://jaeger.dev.storj.io/trace/7dbc99f36bc6674a FAIL (2025-12-15 11:48:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001112845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...................X] duration: 30.378955396s trace: https://jaeger.dev.storj.io/trace/328ec34d2eb5b93c FAIL (2025-12-15 11:48:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 11:48:04.267806751 (236h19m4.138889797s) [upload=207194]: Starting upload Dec 15 11:48:04.267881564 (236h19m4.138964609s) [upload=207194]: Done starting upload Dec 15 11:48:04.267964322 (236h19m4.139047367s) [upload=207194]: Uploading segments... Dec 15 11:48:04.268116571 (236h19m4.139199617s) [upload=207194]: Got next segment. Inline: false Dec 15 11:48:04.268155566 (236h19m4.139238611s) [upload=207194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:48:04.268161216 (236h19m4.139244261s) [upload=207194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:48:04.645157786 (236h19m4.516240831s) [seg_pos={0 0}, upload=207194]: Begin upload segment... Dec 15 11:48:04.645232939 (236h19m4.516315984s) [seg_pos={0 0}, upload=207194]: Done begin upload segment. Dec 15 11:48:04.645355942 (236h19m4.516438988s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.645373907 (236h19m4.516456942s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646387267 (236h19m4.517470312s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.646492347 (236h19m4.517575392s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647115493 (236h19m4.518198549s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.647917941 (236h19m4.519000986s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.648686515 (236h19m4.51976956s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.649098740 (236h19m4.520181795s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650330255 (236h19m4.5214133s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650556706 (236h19m4.521639751s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.650759122 (236h19m4.521842167s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651119207 (236h19m4.522202252s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651345759 (236h19m4.522428804s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651571018 (236h19m4.522654053s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.651948907 (236h19m4.523031952s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:04.652055821 (236h19m4.523138866s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Uploading piece... Dec 15 11:48:05.780921391 (236h19m5.652004436s) [noise=true , address=77.90.31.12:4589 , piece_id=E2NO3HFKIPAHQVQVVOMB2Z75YLPAZIN3SNOQB6YUBYWDAYNU6GRQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.074553880 (236h19m5.945636935s) [noise=true , address=145.79.227.30:4763 , piece_id=KRCKGDK2A3WQLJUDTMOV5EKBIIBJTZWYOFPJBFAOBQDQXBC76UGQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 15 11:48:06.773692793 (236h19m6.644775838s) [upload=207194]: Next returned nil segment Dec 15 11:48:06.773722018 (236h19m6.644805053s) [upload=207194]: Waiting for error group managing segments... Dec 15 11:48:07.197011248 (236h19m7.068094293s) [noise=true , address=145.79.227.21:4474 , piece_id=NGYU5F3B54JOWDHNJSRCSNO5HAEBREECAD3HLHVTL2DJ62IBTNPA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:07.686745571 (236h19m7.557828616s) [noise=true , address=145.79.227.36:4517 , piece_id=V6XL7JD7FTZ2ORL3GO2T3JS2KTY5VABIBFI46LFCZB2XISCQSMYQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:10.935626099 (236h19m10.806709144s) [noise=true , address=145.79.227.10:4401 , piece_id=MJJ2EGE2BD36UON7Q6UJT2JPUCAJNIYM7BJ2SOOU3NYBLWINPDLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.390197256 (236h19m11.261280301s) [noise=true , address=45.91.116.8:4045 , piece_id=A5NBLX7U7NDMM7CX5RLESNMK4EDI6JN6RRDWJLCWBBW5ESOBWA6Q, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.484228721 (236h19m11.355311766s) [noise=true , address=145.79.227.35:4482 , piece_id=G7HDMILSQWUHBVDFW2QN34P3QJFIDUZ3T6UVE4CWEWLECGYGOBLQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:11.850925194 (236h19m11.722008239s) [noise=true , address=145.79.227.46:4602 , piece_id=XXG4I4PJCTPSOAORADKGIB5LK2AVJ3XMDTRSI52IXBDJBD5LKKIA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:15.325714041 (236h19m15.196797086s) [noise=true , address=145.79.227.33:4871 , piece_id=JFEUJM24E4KEWQL72OJMM5XOOMGXK2JOOT2YHODLDWVODMUKXOHQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:16.269216292 (236h19m16.140299337s) [noise=true , address=145.79.227.41:4086 , piece_id=4TOTDLMKOK2VNHGPWTFTD5RQZRRHXDFVE33FSXACCEUWTI6K7ANA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:17.377248139 (236h19m17.248331184s) [noise=true , address=145.79.227.9:4367 , piece_id=45336BRGROQQK6NQ4PTPLGD4VGB3T2WJSRF2GHELBGAVQDQYM3JQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:20.448167344 (236h19m20.319250379s) [noise=true , address=145.79.227.19:4077 , piece_id=WXKBQFOZPQQNIMB5645BZBCZFZPBRR54BFIVWKYAIVLCY2N5LHTA, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:26.357864502 (236h19m26.228947547s) [noise=true , address=145.79.227.50:4375 , piece_id=76WBJI5CE4UN4PVML72HRZJLMQWEYZ6GDQJ6VENLFVT6W5ZAM6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:27.921052250 (236h19m27.792135285s) [noise=true , address=145.79.227.27:4689 , piece_id=3VI2SH4BNBJJ3EXWNUGSTVKMBBBLKX76KZARFYCW7IKTN3IWDQTQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: <nil> Dec 15 11:48:33.166255103 (236h19m33.037338138s) [noise=true , address=145.79.227.29:4942 , piece_id=4JNAQSM36MVGOSVTCHBLY6GKOQFQCF2PEGQLNGOE7FNIIEWNE6BQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166426560 (236h19m33.037510236s) [noise=true , address=145.79.227.8:4329 , piece_id=LYVBZYDSUT2VF4O6BXWWO6I7HKDAIHCOU5QPN6A6UGE2GGTE4MOQ, seg_pos={0 0}, upload=207194]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 11:48:33.166523425 (236h19m33.03760646s) [seg_pos={0 0}, upload=207194]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 11:48:33.542478980 (236h19m33.413562025s) [upload=207194]: Done uploading segments... Dec 15 11:48:33.542516501 (236h19m33.413599536s) [upload=207194]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.378982789s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.............X.............X........X.X.X....X...........X..] duration: 5.001359437s trace: https://jaeger.dev.storj.io/trace/262241722207f9f3 FAIL (2025-12-15 11:48:21 UTC) orbiter warning: context deadline exceeded Duration: 5.001375568s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 12.508155352s trace: https://jaeger.dev.storj.io/trace/10e52dcd32aac5cd FAIL (2025-12-15 11:48:19 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881611B33898F8D, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 12.508215987s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..............X.............X........X.X.X....X...........X.] duration: 5.001785978s trace: https://jaeger.dev.storj.io/trace/262241722207f9f3 FAIL (2025-12-15 11:48:01 UTC) orbiter warning: context deadline exceeded Duration: 5.001816116s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 9.49876798s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:47:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 9.498794512s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...............X.............X........X.X.X....X...........X] duration: 5.001519868s trace: https://jaeger.dev.storj.io/trace/262241722207f9f3 FAIL (2025-12-15 11:47:32 UTC) orbiter warning: context deadline exceeded Duration: 5.001544615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [................X.............X........X.X.X....X...........] duration: 4.322475119s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:47:02 UTC) orbiter warning: context deadline exceeded Duration: 4.322519504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 10.991725125s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:46:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 10.991756013s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 9.400604442s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:46:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 9.400638156s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X................X.............X........X.X.X....X..........] duration: 4.95446615s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:45:59 UTC) orbiter warning: context deadline exceeded Duration: 4.95449205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 19.466248489s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:45:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 19.466277744s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X................X.............X........X.X.X....X.........] duration: 3.852023323s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:45:13 UTC) orbiter warning: context deadline exceeded Duration: 3.852063379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 8.654802842s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:44:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 8.654830385s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X................X.............X........X.X.X....X........] duration: 3.558466905s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:44:10 UTC) orbiter warning: context deadline exceeded Duration: 3.558496872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 9.854439447s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:43:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 9.854465776s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X................X.............X........X.X.X....X.......] duration: 4.52959587s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:43:42 UTC) orbiter warning: context deadline exceeded Duration: 4.529623833s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X................X.............X........X.X.X....X......] duration: 5.00098845s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:43:23 UTC) orbiter warning: context deadline exceeded Duration: 5.001018527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 9.2174535s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:42:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 9.217482335s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X................X.............X........X.X.X....X.....] duration: 4.87940831s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:42:18 UTC) orbiter warning: context deadline exceeded Duration: 4.879455209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 9.018584137s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:41:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 9.01861224s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......X................X.............X........X.X.X....X....] duration: 5.00144033s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:41:22 UTC) orbiter warning: context deadline exceeded Duration: 5.001457161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 9.121657153s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:41:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 9.121685275s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.......X................X.............X........X.X.X....X...] duration: 4.894451307s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:40:55 UTC) orbiter warning: context deadline exceeded Duration: 4.89447879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 11.066943654s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:40:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 11.066991345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 12.242684063s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:39:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 12.242720102s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [........X................X.............X........X.X.X....X..] duration: 4.235849865s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:39:54 UTC) orbiter warning: context deadline exceeded Duration: 4.235878279s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........X................X.............X........X.X.X....X.] duration: 4.745071606s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:38:53 UTC) orbiter warning: context deadline exceeded Duration: 4.745130457s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 8.766763601s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:38:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 8.76693156s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 8.179727105s trace: https://jaeger.dev.storj.io/trace/7f6b0653e2e43d5e FAIL (2025-12-15 11:38:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A) context deadline exceeded ======LOGS====== Dec 15 11:32:22.864660905 (236h3m22.73574395s) [upload=206970]: Starting upload Dec 15 11:32:22.864761717 (236h3m22.735844792s) [upload=206970]: Done starting upload Dec 15 11:32:22.864906533 (236h3m22.735989568s) [upload=206970]: Uploading segments... Dec 15 11:32:22.865007585 (236h3m22.73609062s) [upload=206970]: Got next segment. Inline: false Dec 15 11:32:22.865041630 (236h3m22.736124665s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 11:32:22.865046590 (236h3m22.736129635s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 11:32:23.203598211 (236h3m23.074681256s) [seg_pos={0 0}, upload=206970]: Begin upload segment... Dec 15 11:32:23.203687631 (236h3m23.074770677s) [seg_pos={0 0}, upload=206970]: Done begin upload segment. Dec 15 11:32:23.203759518 (236h3m23.074842563s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.203776821 (236h3m23.074859856s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204408755 (236h3m23.0754918s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.204853611 (236h3m23.075936646s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205208147 (236h3m23.076291182s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.205547753 (236h3m23.076630798s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206041985 (236h3m23.07712503s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.206630997 (236h3m23.077714042s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210145880 (236h3m23.081228915s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.210720603 (236h3m23.081803648s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211113050 (236h3m23.082196095s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.211347677 (236h3m23.082430712s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.212137121 (236h3m23.083220166s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.214715230 (236h3m23.085798285s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.215698592 (236h3m23.086781637s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:23.218626257 (236h3m23.089709302s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Uploading piece... Dec 15 11:32:24.322678770 (236h3m24.193761805s) [noise=true , address=77.90.31.12:4589 , piece_id=PVRUH47IEHOWB63WPCISHIRK6NRDYV2OCKCGN44RNZGQLEAUX2LQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 15 11:32:25.030853650 (236h3m24.901936695s) [upload=206970]: Next returned nil segment Dec 15 11:32:25.030871103 (236h3m24.901954148s) [upload=206970]: Waiting for error group managing segments... Dec 15 11:32:25.383036068 (236h3m25.254119113s) [noise=true , address=77.90.31.11:4555 , piece_id=K7FACVXLYITTJXMXBEMF33UZ3YU3CDYJILRQWRMYRMYCOCEOBHIQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.467838689 (236h3m25.338921734s) [noise=true , address=77.90.31.14:4663 , piece_id=Y4HAAOGY5V734LU2VHISG5NIIFI5JTYOLHD2DVALULKLFRIFUMVQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.590078860 (236h3m25.461161895s) [noise=true , address=145.79.227.33:4871 , piece_id=A5CODLXOMSBEEF6RV7WWRIQSEIMCYQY3CFN7LXXMZXQR2JMUVX4Q, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.596750594 (236h3m25.467833639s) [noise=true , address=145.79.227.43:4158 , piece_id=SO5RRXNIQ3FWTKJQ2NSZYIJ7ZXRJIOY5J4CKR6BAXV4HXOYKEVPA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.636315303 (236h3m25.507398338s) [noise=true , address=145.79.227.36:4517 , piece_id=QSB5HZDMNCMULYXUWZCTWDL4KIOGMY6O2ED3DUYZEZ45EFZA534A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.639671264 (236h3m25.510754309s) [noise=true , address=145.79.227.34:4905 , piece_id=OXFGRGKURFIFCSTNE33TCEA5WV7PKWAU6QU6EIBTJTWVGK7NJXJA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.982742141 (236h3m25.853825176s) [noise=true , address=145.79.227.47:4267 , piece_id=6O2YKORIDZP6JJCEWVXOSGALRPC7F7G2SXP4ZJH4BRPP53TQNE7A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:25.992698129 (236h3m25.863781165s) [noise=true , address=145.79.227.32:4834 , piece_id=CVAIEINFI5YVB36ZKMW4V544D3IVTW2VE243RVO2745Q6GFGZFZQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.046843163 (236h3m25.917926198s) [noise=true , address=77.90.31.10:4519 , piece_id=2VWOIJSGMJSCWPWBCYOFJ4VJ7QYNDARLF6ND4TMIENR266KXAX2A, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.140332665 (236h3m26.01141571s) [noise=true , address=145.79.227.26:4653 , piece_id=BDG4IQJH4ATVCQKGIAV25HHB5UHXNA4CRZBJ7QJSWGNXYQ6W4CHA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.143324422 (236h3m26.014407467s) [noise=true , address=145.79.227.37:4566 , piece_id=BEV7YW2APPMMBDC3A74YKOYZDLQ3SASDPC75CFLWCLVY4KMQNFOA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154691207 (236h3m26.025774252s) [noise=true , address=145.79.227.23:4546 , piece_id=5IVZS5GUPNIYP4XZXBDLG4Z7X72WGQ4MVG4W24SNULKRFO5ESZSA, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154927006 (236h3m26.026010051s) [noise=true , address=145.79.227.25:4618 , piece_id=R3XYOBF52WXXTOPCHD3KYGTMLKE2TAAZRO64MDAR3TTJI7OH2HLQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: <nil> Dec 15 11:32:26.154946643 (236h3m26.026029698s) [seg_pos={0 0}, upload=206970]: Segment reached optimal threshold of 13 pieces. Dec 15 11:32:26.155153507 (236h3m26.026236552s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155160892 (236h3m26.026243947s) [noise=true , address=145.79.227.44:4194 , piece_id=75WDJMKW53S6MW6QVA4UQ6CRQMIMJDONPRGFI4QPOKIQPH7SO4KQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155197652 (236h3m26.026280697s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 11:32:26.155210325 (236h3m26.02629337s) [noise=true , address=145.79.227.38:4978 , piece_id=YLXWDCEDDYSOUAM5D5Q5FC5C35P7BSVQIFSRUZMAN2G7SVKRVIAQ, seg_pos={0 0}, upload=206970]: Long tail cancellation due to optimal threshold Dec 15 11:32:26.155244851 (236h3m26.026327896s) [seg_pos={0 0}, upload=206970]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 11:32:26.155334352 (236h3m26.026417397s) [upload=206970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 11:32:26.155341205 (236h3m26.02642425s) [upload=206970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 11:32:26.545216277 (236h3m26.416299322s) [upload=206970]: Done uploading segments... Dec 15 11:32:26.545243568 (236h3m26.416326614s) [upload=206970]: Upload finished. err: <nil> Duration: 8.179758002s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..........X................X.............X........X.X.X....X] duration: 5.001469494s trace: https://jaeger.dev.storj.io/trace/732959c06c233c3c FAIL (2025-12-15 11:37:45 UTC) orbiter warning: context deadline exceeded Duration: 5.001492507s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)