Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............X......X....X........X........................] duration: 3.500905594s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:55:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.500932117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.703364299s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:55:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.703394667s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............X......X....X........X.......................] duration: 3.510417767s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:55:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510450429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.787490991s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:54:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.787524053s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................X......X....X........X......................] duration: 3.365808524s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:54:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.365835056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.892492414s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:53:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.892523032s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................X......X....X........X.....................] duration: 3.346638331s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:53:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.346661977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.761696944s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:53:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 5.76174196s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X.................X......X....X........X....................] duration: 3.834492864s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:52:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.834518353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.391391023s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:52:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 6.391420799s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X.................X......X....X........X...................] duration: 3.701986857s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:52:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.70201334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003035008s trace: https://jaeger.dev.storj.io/trace/47d93075594603c9 FAIL (2025-12-04 14:52:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: 7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ) context deadline exceeded ======LOGS====== Dec 4 14:51:32.976915811 (21h56m9.44812655s) [upload=19268]: Starting upload Dec 4 14:51:32.976973872 (21h56m9.448184611s) [upload=19268]: Done starting upload Dec 4 14:51:32.977434318 (21h56m9.448645057s) [upload=19268]: Uploading segments... Dec 4 14:51:32.977551711 (21h56m9.448762461s) [upload=19268]: Got next segment. Inline: false Dec 4 14:51:32.977627256 (21h56m9.448837995s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 14:51:32.977634830 (21h56m9.448845569s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 14:51:33.484134649 (21h56m9.955345388s) [seg_pos={0 0}, upload=19268]: Begin upload segment... Dec 4 14:51:33.484197830 (21h56m9.955408569s) [seg_pos={0 0}, upload=19268]: Done begin upload segment. Dec 4 14:51:33.484237866 (21h56m9.955448605s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484248406 (21h56m9.955459165s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484308430 (21h56m9.955519169s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.484737447 (21h56m9.955948186s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485118042 (21h56m9.956328781s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485539714 (21h56m9.956750454s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.485979392 (21h56m9.957190131s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.486397658 (21h56m9.957608398s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.487083014 (21h56m9.958293743s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.488019267 (21h56m9.959230006s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.489329651 (21h56m9.96054039s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.490992358 (21h56m9.962203097s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.492486342 (21h56m9.963697082s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.494071271 (21h56m9.96528201s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.495518957 (21h56m9.966729696s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:33.496915586 (21h56m9.968126325s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Uploading piece... Dec 4 14:51:34.599175730 (21h56m11.07038647s) [noise=true , address=77.90.31.19:4841 , piece_id=7RGHO5ONX3RE5GX3UOCB7ITFAM5Z6A6WTSGKM3JYFUZKSN3AS3LQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 14:51:35.308650010 (21h56m11.779860739s) [upload=19268]: Next returned nil segment Dec 4 14:51:35.308663165 (21h56m11.779873904s) [upload=19268]: Waiting for error group managing segments... Dec 4 14:51:35.594152258 (21h56m12.065362997s) [noise=true , address=77.90.31.14:4661 , piece_id=KEIB4IPIKGT6EEQVS5RRRJFBU4MKASYPBDNX3PAQCULIVUCB2WJA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.648755943 (21h56m12.119966682s) [noise=true , address=145.79.227.27:4689 , piece_id=KYXOH4ERXQEXUOHCC3XNTXWFXW63ZJCVYJ77CYDFX5Y5J5NDNGMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.670976486 (21h56m12.142187225s) [noise=true , address=145.79.227.26:4655 , piece_id=57RUGICR6KHA3EOLOZ3IBXCOG25QS5TECW4SGG6FC6OMOZX7ULUA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.674074645 (21h56m12.145285384s) [noise=true , address=145.79.227.44:4195 , piece_id=DOG5OS3RPBZIYBMS7G664DBG5WX4VMSSN2UO2YOXDHRIE7XPD2OQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.683321492 (21h56m12.154532231s) [noise=true , address=145.79.227.39:4015 , piece_id=JAWTODVFZGNMOI2DWGN7NTXMAGIT7O2YXWX3RJVPQWT2UYAR6KJQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.692392766 (21h56m12.163603505s) [noise=true , address=145.79.227.29:4943 , piece_id=7XTQ37OLKBGPDEIXVCAMU6KXFDY7G6SI25OHZ4LU5LGZJF6D3VZQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.752629928 (21h56m12.223840667s) [noise=true , address=145.79.227.25:4617 , piece_id=5U5AWGVO3XAGQRIGMX35UX54TBZNESDDSRAATJNUSU5NJKQESCNA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:35.924386364 (21h56m12.395597103s) [noise=true , address=145.79.227.10:4401 , piece_id=E6S6E2CQHRFKHNCH46YYFMRDAIMP6FXQR3PX3DGSD7E6EAAE5SDQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.324118818 (21h56m12.795329547s) [noise=true , address=145.79.227.13:4804 , piece_id=XBTE3ETLOA7DD6BVFLHZJMIPRQEANREQLA4HWADARSB5JGPC6DTA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.355704273 (21h56m12.826915002s) [noise=true , address=145.79.227.31:4798 , piece_id=7SMH4Q3JILSNX6VGJB52KKSNGJJ7F4EYO6O5FB3QN7CMIMJPDDIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.373584689 (21h56m12.844795418s) [noise=true , address=145.79.227.9:4367 , piece_id=AVLNCZMNPQBHOSPU2WJLXBOUINBYN3L6TIFAV62OUSOMWK4XDNIQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.668943848 (21h56m13.140154578s) [noise=true , address=145.79.227.32:4835 , piece_id=XGO5UECQU4675MREGVNGU2IVQMZ3GLX3FSECNXWMBJMPQ2XAUWQA, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688768858 (21h56m13.159979587s) [noise=true , address=145.79.227.42:4123 , piece_id=2IBCTUEAGSYSNU3QE3F5BYULMEV327JFPSJRB5FNEFPOUXPB5OMQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: <nil> Dec 4 14:51:36.688789338 (21h56m13.160000067s) [seg_pos={0 0}, upload=19268]: Segment reached optimal threshold of 13 pieces. Dec 4 14:51:36.688991762 (21h56m13.160202491s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Dec 4 14:51:36.688997765 (21h56m13.160208504s) [noise=true , address=145.79.227.22:4510 , piece_id=NGGFWOFVW23FC7WHVK24VPCGDEZEVYAQYKGWNWEUU3GZD2YOU47A, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689426661 (21h56m13.1606374s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: context canceled; context canceled Dec 4 14:51:36.689444414 (21h56m13.160655143s) [noise=true , address=145.79.227.8:4330 , piece_id=SCDD7PKCCFGCB5I6G4GYTELKTOGCYHSGYA4GPRBTTEXBE77IMJNQ, seg_pos={0 0}, upload=19268]: Long tail cancellation due to optimal threshold Dec 4 14:51:36.689510260 (21h56m13.160720999s) [seg_pos={0 0}, upload=19268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 14:51:36.689601073 (21h56m13.160811802s) [upload=19268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 14:51:36.689606934 (21h56m13.160817663s) [upload=19268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 14:51:37.016201901 (21h56m13.48741263s) [upload=19268]: Done uploading segments... Dec 4 14:51:37.016227900 (21h56m13.487438629s) [upload=19268]: Upload finished. err: <nil> Duration: 30.003057391s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X.................X......X....X........X..................] duration: 3.426570616s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:51:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.426627215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X.................X......X....X........X.................] duration: 3.427175159s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:49:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427204166s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....X.................X......X....X........X................] duration: 5.001047869s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:49:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001070543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....X.................X......X....X........X...............] duration: 3.540481948s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:48:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.540513548s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......X.................X......X....X........X..............] duration: 3.713452974s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:46:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.713479616s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.......X.................X......X....X........X.............] duration: 3.607246829s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:46:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.607297144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [........X.................X......X....X........X............] duration: 3.494354294s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:45:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494382266s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.........X.................X......X....X........X...........] duration: 3.506901073s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:44:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.506928646s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..........X.................X......X....X........X..........] duration: 3.385557383s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:44:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.385584345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...........X.................X......X....X........X.........] duration: 3.555564697s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:43:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555591208s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [............X.................X......X....X........X........] duration: 3.484693206s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:42:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484721481s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.............X.................X......X....X........X.......] duration: 3.607127953s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:41:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.607154103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..............X.................X......X....X........X......] duration: 3.462320844s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:41:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46234961s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...............X.................X......X....X........X.....] duration: 5.000711247s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:40:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000736746s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................X.................X......X....X........X....] duration: 3.437226911s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:40:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.437253071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.................X.................X......X....X........X...] duration: 3.85336422s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:39:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.853388797s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..................X.................X......X....X........X..] duration: 3.532970584s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:38:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532989801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...................X.................X......X....X........X.] duration: 3.576919681s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:37:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.576947124s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....................X.................X......X....X........X] duration: 5.001481165s trace: https://jaeger.dev.storj.io/trace/2d68a7ae91d8b315 FAIL (2025-12-04 14:36:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001493328s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....................X.................X......X....X........] duration: 3.824620988s trace: https://jaeger.dev.storj.io/trace/5d6ba17fdbd6b36 FAIL (2025-12-04 14:36:14 UTC) orbiter warning: context deadline exceeded Duration: 3.824652177s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................X.................X......X....X.......] duration: 3.410891595s trace: https://jaeger.dev.storj.io/trace/5d6ba17fdbd6b36 FAIL (2025-12-04 14:35:35 UTC) orbiter warning: context deadline exceeded Duration: 3.410907414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................X.................X......X....X......] duration: 3.516138494s trace: https://jaeger.dev.storj.io/trace/5d6ba17fdbd6b36 FAIL (2025-12-04 14:34:52 UTC) orbiter warning: context deadline exceeded Duration: 3.51616796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................X.................X......X....X.....] duration: 3.514785496s trace: https://jaeger.dev.storj.io/trace/5d6ba17fdbd6b36 FAIL (2025-12-04 14:33:55 UTC) orbiter warning: context deadline exceeded Duration: 3.514913871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........................X.................X......X....X....] duration: 3.478960339s trace: https://jaeger.dev.storj.io/trace/5d6ba17fdbd6b36 FAIL (2025-12-04 14:33:03 UTC) orbiter warning: context deadline exceeded Duration: 3.478994014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........................X.................X......X....X...] duration: 3.686038147s trace: https://jaeger.dev.storj.io/trace/5d6ba17fdbd6b36 FAIL (2025-12-04 14:32:06 UTC) orbiter warning: context deadline exceeded Duration: 3.686062012s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........................X.................X......X....X..] duration: 3.44735652s trace: https://jaeger.dev.storj.io/trace/5d6ba17fdbd6b36 FAIL (2025-12-04 14:31:08 UTC) orbiter warning: context deadline exceeded Duration: 3.447388131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............................X.................X......X....X.] duration: 3.457372625s trace: https://jaeger.dev.storj.io/trace/5d6ba17fdbd6b36 FAIL (2025-12-04 14:30:43 UTC) orbiter warning: context deadline exceeded Duration: 3.457400197s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............................X.................X......X....X] duration: 5.000932783s trace: https://jaeger.dev.storj.io/trace/5d6ba17fdbd6b36 FAIL (2025-12-04 14:29:34 UTC) orbiter warning: context deadline exceeded Duration: 5.000990583s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................X.................X......X....] duration: 4.563862537s trace: https://jaeger.dev.storj.io/trace/cb41ee7f9443592 FAIL (2025-12-04 14:28:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.563890539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................X.................X......X...] duration: 3.594053098s trace: https://jaeger.dev.storj.io/trace/cb41ee7f9443592 FAIL (2025-12-04 14:28:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.594082293s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................X.................X......X..] duration: 3.471069426s trace: https://jaeger.dev.storj.io/trace/cb41ee7f9443592 FAIL (2025-12-04 14:27:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471093813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................X.................X......X.] duration: 3.473918868s trace: https://jaeger.dev.storj.io/trace/cb41ee7f9443592 FAIL (2025-12-04 14:27:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.473949286s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................X.................X......X] duration: 5.002030275s trace: https://jaeger.dev.storj.io/trace/cb41ee7f9443592 FAIL (2025-12-04 14:26:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002077835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X.................X......] duration: 3.611561547s trace: https://jaeger.dev.storj.io/trace/61feaa3186d863f0 FAIL (2025-12-04 14:25:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.611590934s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X.................X.....] duration: 3.406363359s trace: https://jaeger.dev.storj.io/trace/61feaa3186d863f0 FAIL (2025-12-04 14:25:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.406393636s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................X.................X....] duration: 3.556713883s trace: https://jaeger.dev.storj.io/trace/61feaa3186d863f0 FAIL (2025-12-04 14:24:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.556740023s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X.................X...] duration: 3.488341445s trace: https://jaeger.dev.storj.io/trace/61feaa3186d863f0 FAIL (2025-12-04 14:23:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.488369608s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X.................X..] duration: 4.347005507s trace: https://jaeger.dev.storj.io/trace/61feaa3186d863f0 FAIL (2025-12-04 14:22:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.347055732s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)