Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X........X.................................................] duration: 3.426989179s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:15:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427098006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.694162545s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:15:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.694196791s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X........X................................................] duration: 3.475842488s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:15:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.475872544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.714852619s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:14:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.714885352s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X........X...............................................] duration: 3.485367505s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:14:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.485394748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X........X..............................................] duration: 3.45051765s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:13:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.450548519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.809675913s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:13:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.809703496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.766615717s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:13:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.766702642s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X....X........X.............................................] duration: 3.365111024s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:12:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.365138057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.752837418s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:12:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.752869249s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X....X........X............................................] duration: 3.489767932s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:11:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.489816444s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.753006465s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:11:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.753032745s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X....X........X...........................................] duration: 3.516271362s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:10:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516295127s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.630492528s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:10:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.630524958s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.762457997s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:09:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.762495849s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X....X........X..........................................] duration: 3.63431223s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:09:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.634357777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X....X........X.........................................] duration: 3.429677171s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:09:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42973983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X....X........X........................................] duration: 3.43867994s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:08:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.438742681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.806526615s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:08:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.806554237s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X....X........X.......................................] duration: 3.436864656s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:08:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.436895174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.772189477s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:07:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.772219152s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X......X....X........X......................................] duration: 3.424417259s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:07:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42444942s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X......X....X........X.....................................] duration: 3.488774836s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:06:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.488800496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.926181027s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:06:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.926200083s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X......X....X........X....................................] duration: 3.446290016s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:05:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.446314484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.780812043s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:05:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.780840197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.788125434s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:05:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.788197752s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X......X....X........X...................................] duration: 3.492744624s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:04:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492764482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.793685387s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:04:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.79371351s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X......X....X........X..................................] duration: 3.527282081s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:03:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527305665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.757069793s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:03:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.757102084s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X......X....X........X.................................] duration: 3.469880166s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:03:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469928679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.805282709s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:03:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.805311434s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X......X....X........X................................] duration: 3.484711658s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:02:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48473856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.787298255s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:02:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.787390581s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X......X....X........X...............................] duration: 3.40311299s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:01:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.403159719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.748201657s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:01:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.748227537s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X......X....X........X..............................] duration: 3.515309233s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:01:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515341203s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........X......X....X........X.............................] duration: 3.444371344s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 15:00:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444399857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.939945983s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 15:00:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.940336196s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........X......X....X........X............................] duration: 3.503436666s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:59:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503543589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.671375145s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:59:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.671402577s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X......X....X........X...........................] duration: 3.347076324s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:58:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.347104688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.745925513s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:58:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.745959498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.672244261s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:57:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.672276011s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X......X....X........X..........................] duration: 3.542977798s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:57:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.543007404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.734020815s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:56:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.734050442s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X......X....X........X.........................] duration: 3.625126294s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:56:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.625151862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.643122451s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:56:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.643150614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.760268416s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.760293784s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)